Monitor, recover, and upgrade

Monitor

Track hub and provider readiness, reconciliation failures, agent connectivity, and persistent-store health. A provider answering HTTP is not necessarily ready to reconcile resources; use its configured readiness probe and resource conditions.

For an unavailable capability, inspect its pod events and logs, hub registration/heartbeat, and accepted API bindings before restarting components.

Every hub and provider chart serves Prometheus metrics on a separate plain-HTTP listener (metrics.port, default 9090) and annotates its pods with prometheus.io/* scrape annotations, so an annotation-driven collector picks them up without operator CRDs.

Inspect the hosting cluster from the CLI

These commands use the Kubernetes context that hosts the hub/provider pods, not a Railgrid workspace or connected-edge context. Replace HOSTING-CONTEXT, NAMESPACE, POD-NAME, and CONTAINER-NAME with your deployment’s values.

kubectl --context HOSTING-CONTEXT get pods -n NAMESPACE
kubectl --context HOSTING-CONTEXT get deployments,statefulsets -n NAMESPACE
kubectl --context HOSTING-CONTEXT describe pod POD-NAME -n NAMESPACE
kubectl --context HOSTING-CONTEXT logs POD-NAME -n NAMESPACE -c CONTAINER-NAME --tail=100
kubectl --context HOSTING-CONTEXT get events -n NAMESPACE --sort-by=.metadata.creationTimestamp

Use the container names shown by pod inspection. The hub chart’s serving container is hub; provider container names vary. Check readiness, restart counts, probe failures, image pulls, and storage errors. If a container restarted, inspect its previous logs with --previous. Redact credentials and request data before sharing logs.

Check rollout state against the discovered workload name:

kubectl --context HOSTING-CONTEXT rollout status deployment/DEPLOYMENT-NAME -n NAMESPACE --timeout=180s

For embedded-kcp hubs, use statefulset/STATEFULSET-NAME instead. A timeout does not roll back an installation. Inspect events and logs before deciding on a recovery action.

Record the installed Helm release

helm --kube-context HOSTING-CONTEXT list -n NAMESPACE
helm --kube-context HOSTING-CONTEXT status RELEASE-NAME -n NAMESPACE
helm --kube-context HOSTING-CONTEXT history RELEASE-NAME -n NAMESPACE

Record chart and image versions before upgrading. Helm status is not an end-to-end health check: also verify login, workspace API discovery, provider access, and one representative task.

Manage it with GitOps

Railgrid’s own environments are reconciled by Flux : an OCI HelmRepository for oci://ghcr.io/railgrid/charts, one HelmRelease per component with chart.spec.version: "0.2.x", and values in ConfigMaps generated from files in git. Two settings matter when following a release line automatically:

• Set install.remediation.retries: -1 (and a few upgrade retries). The release pipeline can publish a chart minutes before its image finishes pushing; without retries one ImagePullBackOff window stalls the release until the next chart version.
• Keep credentials out of git. Create Secrets imperatively before the release that references them merges; a HelmRelease whose pod mounts a missing Secret stalls until it exists.

Install Flux with flux install and a read-only deploy key rather than flux bootstrap if your repository policy forbids pushes to the default branch; bootstrap commits its manifests directly.

Storage and recovery

Inventory embedded/external kcp state, provider databases, persistent volumes, encryption keys, credentials, and runtime data. Repositories and upstream services have separate retention responsibilities.

Before upgrading, take backups using each datastore’s supported procedure and verify a restore in an isolated environment. Keep encryption keys with the recovery plan: a database backup alone may not make encrypted messages readable. Reconcile runtime resources against restored control-plane state before allowing writes.

This documentation does not establish a tested backup product or recovery-time objective. Define and rehearse a deployment-specific restore procedure before depending on recovery.

Certificates and trust

• The hub reads its serving certificate once, at start. After cert-manager renews the Secret, restart the hub, or issue the certificate with a long duration so that an ordinary release rollout inside the renewal window picks it up (see Install the hub ).
• Minted provider kubeconfigs embed the provider CA bundle. After changing hub.providerCASecretName’s contents, restart the hub and re-copy every provider kubeconfig Secret; see Refresh a provider kubeconfig .
• The hub’s kcp kubeconfig expires. A Kubeconfig minted by the kcp-operator has a validity; re-mint and replace the hub’s Secret before it lapses.
• Published apps need a wildcard. DNS and a certificate for *.<apps domain> must exist before the first app is published, or new URLs fail TLS.

Capacity

A kcp shard pod requests one CPU and runs two replicas per shard, so a two-shard control plane holds four CPUs before any provider runs. A rolling update of a shard needs one more CPU free somewhere. On an autoscaled node pool, leave headroom or a second node pool on a different machine family: a rollout waiting on a node that the cloud cannot provide (capacity stockout in a zone) stays Pending while the old pods keep serving, which blocks every later step.

Scaling

Embedded kcp remains single-replica. External kcp permits multiple hub replicas; the hub’s singleton controllers gate on a Lease and sessions live in kcp-backed Secrets. Provider limits differ: Code needs an S3 bucket for more than one replica, Kuery needs Postgres, Edges and Infrastructure scale freely, and App Studio depends on its chart version. Consult each provider’s hosting guide . Do not apply one replica count to every component.

Upgrade and rollback

1. Record deployed chart, image, and schema versions and read their change notes.
2. Back up required state and preserve the current values and credentials.
3. Rehearse the upgrade against restored/test data.
4. Upgrade compatible components, then verify readiness, sign-in, provider enablement, and a representative task.
5. If it fails, inspect migration compatibility before rolling back. A Helm rollback cannot undo an incompatible database or resource-schema migration.

Order matters between kcp and the providers. kcp and the infrastructure provider move in lockstep (see External kcp ); when following 0.2.x automatically, do the kcp step right around the provider release that needs it. Roll the root shard first: it also hosts the cache server the other shards read from.

A provider that reports Ready=False right after an upgrade with APIExportEndpointSlice publishes no endpoints yet is waiting for a workspace to bind it, not failing; see Provider hosting .

For uninstall, inventory and remove dependent resources first. Helm removal may retain PVCs or secrets; delete retained state only after its owners agree it is no longer needed.