# Monitor, recover, and upgrade

Plan around the state each component owns, keep trust material valid, and upgrade in the right order.

Source: https://railgrid.ai/docs/self-hosting/hub/operations/


## Monitor

Track hub and provider readiness, reconciliation failures, agent connectivity, and persistent-store health. A provider answering HTTP is not necessarily ready to reconcile resources; use its configured readiness probe and resource conditions.

For an unavailable capability, inspect its pod events and logs, hub registration/heartbeat, and accepted API bindings before restarting components.

Every hub and provider chart serves Prometheus metrics on a separate plain-HTTP listener (`metrics.port`, default `9090`) and annotates its pods with `prometheus.io/*` scrape annotations, so an annotation-driven collector picks them up without operator CRDs.

## Inspect the hosting cluster from the CLI

These commands use the Kubernetes context that hosts the hub/provider pods, **not a Railgrid workspace or connected-edge context**. Replace `HOSTING-CONTEXT`, `NAMESPACE`, `POD-NAME`, and `CONTAINER-NAME` with your deployment's values.

```bash
kubectl --context HOSTING-CONTEXT get pods -n NAMESPACE
kubectl --context HOSTING-CONTEXT get deployments,statefulsets -n NAMESPACE
kubectl --context HOSTING-CONTEXT describe pod POD-NAME -n NAMESPACE
kubectl --context HOSTING-CONTEXT logs POD-NAME -n NAMESPACE -c CONTAINER-NAME --tail=100
kubectl --context HOSTING-CONTEXT get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
```

Use the container names shown by pod inspection. The hub chart's serving container is `hub`; provider container names vary. Check readiness, restart counts, probe failures, image pulls, and storage errors. If a container restarted, inspect its previous logs with `--previous`. Redact credentials and request data before sharing logs.

Check rollout state against the discovered workload name:

```bash
kubectl --context HOSTING-CONTEXT rollout status deployment/DEPLOYMENT-NAME -n NAMESPACE --timeout=180s
```

For embedded-kcp hubs, use `statefulset/STATEFULSET-NAME` instead. A timeout does not roll back an installation. Inspect events and logs before deciding on a recovery action.

## Record the installed Helm release

```bash
helm --kube-context HOSTING-CONTEXT list -n NAMESPACE
helm --kube-context HOSTING-CONTEXT status RELEASE-NAME -n NAMESPACE
helm --kube-context HOSTING-CONTEXT history RELEASE-NAME -n NAMESPACE
```

Record chart and image versions before upgrading. Helm status is not an end-to-end health check: also verify login, workspace API discovery, provider access, and one representative task.

## Manage it with GitOps

Railgrid's own environments are reconciled by [Flux](https://fluxcd.io/): an OCI `HelmRepository` for `oci://ghcr.io/railgrid/charts`, one `HelmRelease` per component with `chart.spec.version: "0.2.x"`, and values in ConfigMaps generated from files in git. Two settings matter when following a release line automatically:

- Set `install.remediation.retries: -1` (and a few upgrade retries). The release pipeline can publish a chart minutes before its image finishes pushing; without retries one `ImagePullBackOff` window stalls the release until the next chart version.
- Keep credentials out of git. Create Secrets imperatively before the release that references them merges; a `HelmRelease` whose pod mounts a missing Secret stalls until it exists.

Install Flux with `flux install` and a read-only deploy key rather than `flux bootstrap` if your repository policy forbids pushes to the default branch; bootstrap commits its manifests directly.

## Storage and recovery

Inventory embedded/external kcp state, provider databases, persistent volumes, encryption keys, credentials, and runtime data. Repositories and upstream services have separate retention responsibilities.

| State | Where | Backup |
|---|---|---|
| Workspaces, bindings, every provider's API objects | kcp etcd (embedded PVC, or one etcd per external shard) | etcd snapshots on a schedule to storage outside the cluster |
| Hub serving certificate and CA | cert-manager Secrets in the hub namespace | Back up the CA: rotating it means re-minting every provider kubeconfig |
| Provider data | Postgres (Kuery, App Studio, Agents), object storage (Code bundles) | The database's or bucket's own procedure |
| Dex state | Kubernetes resources or SQLite | Losing it only signs everyone out |

Before upgrading, take backups using each datastore's supported procedure and verify a restore in an isolated environment. Keep encryption keys with the recovery plan: a database backup alone may not make encrypted messages readable. Reconcile runtime resources against restored control-plane state before allowing writes.

This documentation does not establish a tested backup product or recovery-time objective. Define and rehearse a deployment-specific restore procedure before depending on recovery.

## Certificates and trust

- **The hub reads its serving certificate once, at start.** After cert-manager renews the Secret, restart the hub, or issue the certificate with a long duration so that an ordinary release rollout inside the renewal window picks it up (see [Install the hub](https://railgrid.ai/docs/self-hosting/hub/helm/#serving-certificate-and-provider-trust)).
- **Minted provider kubeconfigs embed the provider CA bundle.** After changing `hub.providerCASecretName`'s contents, restart the hub and re-copy every provider kubeconfig Secret; see [Refresh a provider kubeconfig](https://railgrid.ai/docs/self-hosting/providers/#refresh-a-provider-kubeconfig).
- **The hub's kcp kubeconfig expires.** A `Kubeconfig` minted by the kcp-operator has a `validity`; re-mint and replace the hub's Secret before it lapses.
- **Published apps need a wildcard.** DNS and a certificate for `*.<apps domain>` must exist before the first app is published, or new URLs fail TLS.

## Capacity

A kcp shard pod requests one CPU and runs two replicas per shard, so a two-shard control plane holds four CPUs before any provider runs. A rolling update of a shard needs one more CPU free somewhere. On an autoscaled node pool, leave headroom or a second node pool on a different machine family: a rollout waiting on a node that the cloud cannot provide (capacity stockout in a zone) stays Pending while the old pods keep serving, which blocks every later step.

## Scaling

Embedded kcp remains single-replica. External kcp permits multiple hub replicas; the hub's singleton controllers gate on a Lease and sessions live in kcp-backed Secrets. Provider limits differ: Code needs an S3 bucket for more than one replica, Kuery needs Postgres, Edges and Infrastructure scale freely, and App Studio depends on its chart version. Consult each provider's [hosting guide](https://railgrid.ai/docs/self-hosting/providers/). Do not apply one replica count to every component.

## Upgrade and rollback

1. Record deployed chart, image, and schema versions and read their change notes.
2. Back up required state and preserve the current values and credentials.
3. Rehearse the upgrade against restored/test data.
4. Upgrade compatible components, then verify readiness, sign-in, provider enablement, and a representative task.
5. If it fails, inspect migration compatibility before rolling back. A Helm rollback cannot undo an incompatible database or resource-schema migration.

Order matters between kcp and the providers. kcp and the infrastructure provider move in lockstep (see [External kcp](https://railgrid.ai/docs/self-hosting/hub/external-kcp/#versions-move-in-lockstep)); when following `0.2.x` automatically, do the kcp step right around the provider release that needs it. Roll the root shard first: it also hosts the cache server the other shards read from.

A provider that reports `Ready=False` right after an upgrade with `APIExportEndpointSlice publishes no endpoints yet` is waiting for a workspace to bind it, not failing; see [Provider hosting](https://railgrid.ai/docs/self-hosting/providers/#a-fresh-provider-is-unready-until-a-workspace-binds-it).

For uninstall, inventory and remove dependent resources first. Helm removal may retain PVCs or secrets; delete retained state only after its owners agree it is no longer needed.