# External kcp with the kcp-operator

Run kcp shards, etcd, and the front proxy outside the hub pod and connect the hub to them.

Source: https://railgrid.ai/docs/self-hosting/hub/external-kcp/


Run kcp as its own set of workloads so the hub can scale past one replica and the control plane has a lifecycle of its own. This page describes the shape Railgrid runs in both of its environments: the [kcp-operator](https://github.com/kcp-dev/kcp-operator) managing a root shard, one additional shard, and a front proxy, each shard on its own etcd.

The product repo's [external-kcp recipe](https://github.com/railgrid/railgrid/blob/main/docs/install-external-kcp.md) runs the same layout against kind in CI with Envoy Gateway. Use it to see every manifest in one place; this page explains the choices that matter on a real cluster.

## Prerequisites

- A cluster you administer, with [cert-manager](https://cert-manager.io/) installed. The operator issues every kcp certificate through it.
- The [ingress](https://railgrid.ai/docs/self-hosting/hub/ingress/) that will serve the kcp hostnames, with TLS.
- An OIDC issuer URL and client ID: the same values the hub uses.
- Hostnames for the front proxy and for each shard, for example `kcp.example.com`, `kcp-root.example.com`, `kcp-alpha.example.com`.

Install the operator from its Helm chart. The chart ships its CRDs as templates, so `helm upgrade` updates them too:

```bash
helm repo add kcp https://kcp-dev.github.io/helm-charts
helm upgrade --install kcp-operator kcp/kcp-operator \
  -n kcp-operator --create-namespace --version 0.7.10 --set image.tag=v0.10.0
```

Railgrid 0.2.x runs chart `0.7.10` with operator image `v0.10.0`. The operator installs a `RootShard`/`Shard`/`FrontProxy`/`Kubeconfig` API in the `operator.kcp.io/v1alpha1` group.

## Versions move in lockstep

Pin every kcp image explicitly, including the root shard's proxy image; otherwise a component follows the operator's default and drifts on the next operator upgrade. Railgrid 0.2.x is built against kcp 0.33 (`kcp/sdk v0.33.0` in `go.mod`). The infrastructure provider in particular uses kcp kinds that were renamed in 0.33 without automatic migration, so upgrade kcp and the infrastructure provider together and read both sets of release notes first.

## etcd

Each shard needs its own etcd. A single member per shard is enough for a small cluster; for production, run three members per shard and spread them across zones. Two details decided the outcome in practice:

- Add `minDomains: 3` to the topology spread constraint. When the node pool starts with nodes in one zone only, the scheduler counts only zones that have nodes and packs all three members into that zone, where their zonal volumes then pin them permanently. With `minDomains`, the members for empty zones stay Pending until the autoscaler adds a node there.
- Snapshot etcd on a schedule to object storage outside the cluster, and prove one restore before relying on it.

```yaml
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    minDomains: 3
    labelSelector:
      matchLabels:
        app: etcd-root
```

Shards list all members in `spec.etcd.endpoints`. Plain HTTP on `2379` inside the cluster is what the recipe uses; keep etcd unreachable from outside.

## Shards and the front proxy

The operator needs a self-signed `Issuer` in its namespace to build kcp's CAs from. Then apply the root shard, wait for it, and apply the other shard and the front proxy. A trimmed root shard:

```yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: kcp-system
spec:
  selfSigned: {}
---
apiVersion: operator.kcp.io/v1alpha1
kind: RootShard
metadata:
  name: root
  namespace: kcp-system
spec:
  image:
    repository: ghcr.io/kcp-dev/kcp
    tag: v0.33.0            # pin; same tag on the proxy, the other shards and the front proxy
  proxy:
    image:
      repository: ghcr.io/kcp-dev/kcp
      tag: v0.33.0
  replicas: 2
  extraArgs:
    - --feature-gates=CacheAPIs=true
    # What APIExportEndpointSlices advertise. Set THIS, never spec.shardBaseURL.
    - --shard-virtual-workspace-url=https://kcp-root.example.com
  certificateTemplates:
    server:
      spec:
        dnsNames:
          - kcp-root.example.com   # the in-cluster path presents this cert too
  auth:
    serviceAccount:
      enabled: true
    oidc:
      issuerURL: https://auth.example.com
      clientID: railgrid
      groupsClaim: groups
      usernameClaim: email
      usernamePrefix: "railgrid:"
      groupsPrefix: "railgrid:"
  external:
    hostname: frontproxy-front-proxy.kcp-system.svc.cluster.local
    port: 6443
  certificates:
    issuerRef:
      group: cert-manager.io
      kind: Issuer
      name: selfsigned
  cache:
    embedded:
      enabled: true
  etcd:
    endpoints:
      - http://etcd-root-0.etcd-root-headless.etcd.svc.cluster.local:2379
      - http://etcd-root-1.etcd-root-headless.etcd.svc.cluster.local:2379
      - http://etcd-root-2.etcd-root-headless.etcd.svc.cluster.local:2379
```

A `Shard` carries the same `image`, `extraArgs` (with its own virtual-workspace URL), `certificateTemplates`, `auth` and `etcd` blocks, plus `rootShard.ref.name: root`. The `FrontProxy` carries `image`, `replicas`, `external.hostname`/`port`, the same `auth.oidc` block, and `rootShard.ref.name: root`.

Three rules from these manifests:

**Set the virtual-workspace URL, never the base URL.** A shard has three URLs. `--shard-base-url` drives front-proxy routing and the cluster index the authenticator resolves workspaces through; `--shard-external-url` is what gets written into kubeconfigs; `--shard-virtual-workspace-url` is what APIExportEndpointSlices advertise and falls back to the base URL when unset. Pointing `spec.shardBaseURL` at the public name takes the whole platform down within seconds of the shards rolling: every request arrives as an empty user, providers vanish from the console, and sign-in fails. Only the virtual-workspace URL belongs on a public hostname.

**OIDC must match the hub exactly.** The hub hard-codes every user's RBAC identity as `railgrid:<email>`, so `usernameClaim: email` and the `railgrid:` prefixes are not optional. `issuerURL` and `clientID` must equal the hub's `idp.issuerURL` and `idp.clientID`. If the issuer's certificate is not in the kcp image's trust store (a cloud-managed certificate at the edge, for example), give the front proxy the issuer's CA chain with `spec.caBundleSecretRef`.

**Enable `CacheAPIs`.** The infrastructure provider publishes its templates through kcp's cached-resource APIs, which sit behind this feature gate.

Until Dex is up, the shards log failed discovery of the issuer. That is expected at this stage.

## Expose kcp

Two kinds of client reach the shards, and they need different paths.

- **Users and edges** come from outside with OIDC tokens. Route the front-proxy hostname and each shard hostname through your ingress to the operator's Services. Terminating TLS at the edge is fine for them.
- **The hub and in-cluster providers** authenticate to the shards with client certificates, and they follow the public shard URLs that the endpoint slices advertise. If those names resolve to your edge, TLS termination strips the client certificate and every request fails with 401. Inside the cluster, resolve each shard hostname to its Service instead: a CoreDNS `rewrite` on k3s, or a Cloud DNS response policy on GKE (which runs kube-dns). Give the shard Services fixed ClusterIPs so the DNS answer stays valid across re-creates.

Own the Services in front of the operator's pods rather than relying on the operator's, so fields such as a fixed `clusterIP` or `appProtocol: HTTPS` (needed for [Gateway API](https://railgrid.ai/docs/self-hosting/hub/gateway-api/) re-encryption) stick. Select pods with the operator's labels, for example `app.kubernetes.io/component: rootshard` and `app.kubernetes.io/instance: root`.

Give every kcp backend a long request timeout at the load balancer. Watches and edge tunnels are long-lived; a 30-second default closes them.

## Mint the hub's kubeconfig

The operator mints admin kubeconfigs from a `Kubeconfig` object. The hub needs one that goes through the front proxy:

```yaml
apiVersion: operator.kcp.io/v1alpha1
kind: Kubeconfig
metadata:
  name: kcp-kubeconfig-frontproxy
  namespace: kcp-system
spec:
  username: kcp-admin
  groups:
    - system:kcp:admin
  validity: 8766h
  secretRef:
    name: kcp-kubeconfig-frontproxy
  target:
    frontProxyRef:
      name: frontproxy
```

Copy it into the hub namespace under the key the hub chart expects:

```bash
kubectl -n kcp-system get secret kcp-kubeconfig-frontproxy -o jsonpath='{.data.kubeconfig}' \
  | base64 -d \
  | kubectl -n railgrid-system create secret generic railgrid-kcp --from-file=admin.kubeconfig=/dev/stdin
```

The server inside it is the in-cluster front-proxy Service, so the hub never leaves the cluster to reach kcp. It is a `system:kcp:admin` credential; it expires after `validity`, so plan to re-mint it.

## Verify

```bash
kubectl -n kcp-system rollout status deploy/root-kcp deploy/alpha-shard-kcp deploy/frontproxy-front-proxy

# Split horizon: inside a pod a shard name resolves to its Service, not the edge.
kubectl run dnscheck --rm -i --restart=Never --image=curlimages/curl:8.10.1 \
  --command -- nslookup kcp-root.example.com

# Through the ingress. Anonymous /api is a 403 for system:anonymous on a shard
# (healthy) and a 401 on the front proxy, which rejects unauthenticated requests.
curl -s -o /dev/null -w '%{http_code}\n' https://kcp.example.com/readyz    # 200
curl -s https://kcp-root.example.com/api | head -c 120                      # 403 ... system:anonymous
curl -s https://kcp.example.com/api | head -c 120                           # 401
```

For administrative access from a laptop, extract the front-proxy kubeconfig and port-forward the front-proxy Service; its server address is in-cluster. The front proxy advertises only its own client CA in the TLS handshake, which makes `kubectl` send no client certificate at all; `curl --cert --key` authenticates fine. Keep that credential out of version control.

## Next

Install [Dex](https://railgrid.ai/docs/self-hosting/hub/oidc/), then the [hub](https://railgrid.ai/docs/self-hosting/hub/helm/#configure-a-production-hub) with `kcp.external.existingSecret: railgrid-kcp`. Remember to add kcp's `root-server-ca` to the hub's [provider CA bundle](https://railgrid.ai/docs/self-hosting/hub/helm/#serving-certificate-and-provider-trust).