External kcp with the kcp-operator

Run kcp as its own set of workloads so the hub can scale past one replica and the control plane has a lifecycle of its own. This page describes the shape Railgrid runs in both of its environments: the kcp-operator managing a root shard, one additional shard, and a front proxy, each shard on its own etcd.

The product repo’s external-kcp recipe runs the same layout against kind in CI with Envoy Gateway. Use it to see every manifest in one place; this page explains the choices that matter on a real cluster.

Prerequisites

• A cluster you administer, with cert-manager installed. The operator issues every kcp certificate through it.
• The ingress that will serve the kcp hostnames, with TLS.
• An OIDC issuer URL and client ID: the same values the hub uses.
• Hostnames for the front proxy and for each shard, for example kcp.example.com, kcp-root.example.com, kcp-alpha.example.com.

Install the operator from its Helm chart. The chart ships its CRDs as templates, so helm upgrade updates them too:

helm repo add kcp https://kcp-dev.github.io/helm-charts
helm upgrade --install kcp-operator kcp/kcp-operator \
  -n kcp-operator --create-namespace --version 0.7.10 --set image.tag=v0.10.0

Railgrid 0.2.x runs chart 0.7.10 with operator image v0.10.0. The operator installs a RootShard/Shard/FrontProxy/Kubeconfig API in the operator.kcp.io/v1alpha1 group.

Versions move in lockstep

Pin every kcp image explicitly, including the root shard’s proxy image; otherwise a component follows the operator’s default and drifts on the next operator upgrade. Railgrid 0.2.x is built against kcp 0.33 (kcp/sdk v0.33.0 in go.mod). The infrastructure provider in particular uses kcp kinds that were renamed in 0.33 without automatic migration, so upgrade kcp and the infrastructure provider together and read both sets of release notes first.

etcd

Each shard needs its own etcd. A single member per shard is enough for a small cluster; for production, run three members per shard and spread them across zones. Two details decided the outcome in practice:

• Add minDomains: 3 to the topology spread constraint. When the node pool starts with nodes in one zone only, the scheduler counts only zones that have nodes and packs all three members into that zone, where their zonal volumes then pin them permanently. With minDomains, the members for empty zones stay Pending until the autoscaler adds a node there.
• Snapshot etcd on a schedule to object storage outside the cluster, and prove one restore before relying on it.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    minDomains: 3
    labelSelector:
      matchLabels:
        app: etcd-root

Shards list all members in spec.etcd.endpoints. Plain HTTP on 2379 inside the cluster is what the recipe uses; keep etcd unreachable from outside.

Shards and the front proxy

The operator needs a self-signed Issuer in its namespace to build kcp’s CAs from. Then apply the root shard, wait for it, and apply the other shard and the front proxy. A trimmed root shard:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: kcp-system
spec:
  selfSigned: {}
---
apiVersion: operator.kcp.io/v1alpha1
kind: RootShard
metadata:
  name: root
  namespace: kcp-system
spec:
  image:
    repository: ghcr.io/kcp-dev/kcp
    tag: v0.33.0            # pin; same tag on the proxy, the other shards and the front proxy
  proxy:
    image:
      repository: ghcr.io/kcp-dev/kcp
      tag: v0.33.0
  replicas: 2
  extraArgs:
    - --feature-gates=CacheAPIs=true
    # What APIExportEndpointSlices advertise. Set THIS, never spec.shardBaseURL.
    - --shard-virtual-workspace-url=https://kcp-root.example.com
  certificateTemplates:
    server:
      spec:
        dnsNames:
          - kcp-root.example.com   # the in-cluster path presents this cert too
  auth:
    serviceAccount:
      enabled: true
    oidc:
      issuerURL: https://auth.example.com
      clientID: railgrid
      groupsClaim: groups
      usernameClaim: email
      usernamePrefix: "railgrid:"
      groupsPrefix: "railgrid:"
  external:
    hostname: frontproxy-front-proxy.kcp-system.svc.cluster.local
    port: 6443
  certificates:
    issuerRef:
      group: cert-manager.io
      kind: Issuer
      name: selfsigned
  cache:
    embedded:
      enabled: true
  etcd:
    endpoints:
      - http://etcd-root-0.etcd-root-headless.etcd.svc.cluster.local:2379
      - http://etcd-root-1.etcd-root-headless.etcd.svc.cluster.local:2379
      - http://etcd-root-2.etcd-root-headless.etcd.svc.cluster.local:2379

A Shard carries the same image, extraArgs (with its own virtual-workspace URL), certificateTemplates, auth and etcd blocks, plus rootShard.ref.name: root. The FrontProxy carries image, replicas, external.hostname/port, the same auth.oidc block, and rootShard.ref.name: root.

Three rules from these manifests:

Set the virtual-workspace URL, never the base URL. A shard has three URLs. --shard-base-url drives front-proxy routing and the cluster index the authenticator resolves workspaces through; --shard-external-url is what gets written into kubeconfigs; --shard-virtual-workspace-url is what APIExportEndpointSlices advertise and falls back to the base URL when unset. Pointing spec.shardBaseURL at the public name takes the whole platform down within seconds of the shards rolling: every request arrives as an empty user, providers vanish from the console, and sign-in fails. Only the virtual-workspace URL belongs on a public hostname.

OIDC must match the hub exactly. The hub hard-codes every user’s RBAC identity as railgrid:<email>, so usernameClaim: email and the railgrid: prefixes are not optional. issuerURL and clientID must equal the hub’s idp.issuerURL and idp.clientID. If the issuer’s certificate is not in the kcp image’s trust store (a cloud-managed certificate at the edge, for example), give the front proxy the issuer’s CA chain with spec.caBundleSecretRef.

Enable CacheAPIs. The infrastructure provider publishes its templates through kcp’s cached-resource APIs, which sit behind this feature gate.

Until Dex is up, the shards log failed discovery of the issuer. That is expected at this stage.

Expose kcp

Two kinds of client reach the shards, and they need different paths.

• Users and edges come from outside with OIDC tokens. Route the front-proxy hostname and each shard hostname through your ingress to the operator’s Services. Terminating TLS at the edge is fine for them.
• The hub and in-cluster providers authenticate to the shards with client certificates, and they follow the public shard URLs that the endpoint slices advertise. If those names resolve to your edge, TLS termination strips the client certificate and every request fails with 401. Inside the cluster, resolve each shard hostname to its Service instead: a CoreDNS rewrite on k3s, or a Cloud DNS response policy on GKE (which runs kube-dns). Give the shard Services fixed ClusterIPs so the DNS answer stays valid across re-creates.

Own the Services in front of the operator’s pods rather than relying on the operator’s, so fields such as a fixed clusterIP or appProtocol: HTTPS (needed for Gateway API re-encryption) stick. Select pods with the operator’s labels, for example app.kubernetes.io/component: rootshard and app.kubernetes.io/instance: root.

Give every kcp backend a long request timeout at the load balancer. Watches and edge tunnels are long-lived; a 30-second default closes them.

Mint the hub’s kubeconfig

The operator mints admin kubeconfigs from a Kubeconfig object. The hub needs one that goes through the front proxy:

apiVersion: operator.kcp.io/v1alpha1
kind: Kubeconfig
metadata:
  name: kcp-kubeconfig-frontproxy
  namespace: kcp-system
spec:
  username: kcp-admin
  groups:
    - system:kcp:admin
  validity: 8766h
  secretRef:
    name: kcp-kubeconfig-frontproxy
  target:
    frontProxyRef:
      name: frontproxy

Copy it into the hub namespace under the key the hub chart expects:

kubectl -n kcp-system get secret kcp-kubeconfig-frontproxy -o jsonpath='{.data.kubeconfig}' \
  | base64 -d \
  | kubectl -n railgrid-system create secret generic railgrid-kcp --from-file=admin.kubeconfig=/dev/stdin

The server inside it is the in-cluster front-proxy Service, so the hub never leaves the cluster to reach kcp. It is a system:kcp:admin credential; it expires after validity, so plan to re-mint it.

Verify

kubectl -n kcp-system rollout status deploy/root-kcp deploy/alpha-shard-kcp deploy/frontproxy-front-proxy

# Split horizon: inside a pod a shard name resolves to its Service, not the edge.
kubectl run dnscheck --rm -i --restart=Never --image=curlimages/curl:8.10.1 \
  --command -- nslookup kcp-root.example.com

# Through the ingress. Anonymous /api is a 403 for system:anonymous on a shard
# (healthy) and a 401 on the front proxy, which rejects unauthenticated requests.
curl -s -o /dev/null -w '%{http_code}\n' https://kcp.example.com/readyz    # 200
curl -s https://kcp-root.example.com/api | head -c 120                      # 403 ... system:anonymous
curl -s https://kcp.example.com/api | head -c 120                           # 401

For administrative access from a laptop, extract the front-proxy kubeconfig and port-forward the front-proxy Service; its server address is in-cluster. The front proxy advertises only its own client CA in the TLS handshake, which makes kubectl send no client certificate at all; curl --cert --key authenticates fine. Keep that credential out of version control.

Next

Install Dex , then the hub with kcp.external.existingSecret: railgrid-kcp. Remember to add kcp’s root-server-ca to the hub’s provider CA bundle .