vault-platform ClusterSecretStore

The single cluster-wide ClusterSecretStore on each v6 cluster: Vault Kubernetes auth wiring, CA bundle handling, the per-cluster role and policy, and the smoke-test path.

This page documents ClusterSecretStore/vault-platform — the single cluster-wide secret-store CR that platform/operator namespaces use to read Vault. Tenant namespaces never use this CR; they have their own namespace-scoped SecretStore (see 04-tenant-secretstore-pattern).

What it is

A ClusterSecretStore (CSS) is a cluster-scoped CR that declares an authenticated connection to an external secrets system, in our case Vault. Once it’s Ready=True, any ExternalSecret in any namespace that references it with secretStoreRef.kind: ClusterSecretStore, name: vault-platform can read from Vault under the policy attached to its role.

We have exactly one CSS per cluster:

The two CSS objects have the same name on both clusters; the role in their auth.kubernetes block differs so each cluster gets its own least-privilege Vault token.

The manifest

clusters/<cluster>/secrets/eso/clustersecretstore-vault.yaml:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-platform
spec:
  provider:
    vault:
      server: https://vault.sub.comptech-lab.com:8200
      path: secret           # the kv-v2 mount
      version: v2
      caBundle: <base64-encoded Vault CA cert>
      auth:
        kubernetes:
          mountPath: kubernetes-<CLUSTER>     # e.g. kubernetes-spoke-dc-v6
          role: ocp-<CLUSTER>-eso             # e.g. ocp-spoke-dc-v6-eso
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
            audiences:
              - vault

Key fields:

• provider.vault.server — the public-facing Vault URL. Resolves via PowerDNS (pdns.local recursor); resolution does not need internet egress.
• provider.vault.path — the kv-v2 mount, fixed at secret. The path inside the mount is supplied per-ExternalSecret.
• provider.vault.version: v2 — required; tells ESO to use kv-v2’s data/ and metadata/ sub-paths internally.
• caBundle — base64-encoded Vault server cert. Must be a self-signed-friendly bundle because the lab Vault VM is not in any public trust chain.
• auth.kubernetes.mountPath — the cluster-specific Vault Kubernetes auth mount. There is one mount per cluster.
• auth.kubernetes.role — the cluster-specific Vault role with policy ocp-<cluster>-eso.
• auth.kubernetes.serviceAccountRef — the operand SA in external-secrets. The TokenReview against this SA’s JWT is how Vault validates the request.
• audiences: ["vault"] — projected token audience. Must match the Vault role’s audience.

Vault-side wiring

The Vault VM has a separate Kubernetes auth mount per cluster. The naming convention is kubernetes-<cluster> so cross-cluster trust is impossible: hub-dc-v6 ESO authenticates only against auth/kubernetes-hub-dc-v6, never against auth/kubernetes-spoke-dc-v6.

The mount is configured with:

vault auth enable -path=kubernetes-<cluster> kubernetes

vault write auth/kubernetes-<cluster>/config \
  kubernetes_host="https://api.<cluster>.sub.comptech-lab.com:6443" \
  kubernetes_ca_cert="@<path-to-cluster-CA>"  \
  disable_iss_validation=true

The cluster’s API server CA is exported from oc get cm kube-root-ca.crt -n default -o jsonpath='{.data.ca\.crt}'.

The role:

vault write auth/kubernetes-<cluster>/role/ocp-<cluster>-eso \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  token_policies=ocp-<cluster>-eso \
  token_ttl=1h \
  token_max_ttl=4h \
  audience=vault

The policy:

path "secret/data/ocp/<cluster>/*" {
  capabilities = ["read"]
}
path "secret/data/ocp/platform/*" {
  capabilities = ["read"]
}
path "secret/metadata/ocp/<cluster>/*" {
  capabilities = ["list", "read"]
}
path "secret/metadata/ocp/platform/*" {
  capabilities = ["list", "read"]
}

The policy is intentionally read-only and path-pinned. The vault-platform CSS cannot:

• Write any secret.
• Read tenant paths under secret/data/apps/... — those belong to tenant SecretStores.
• Read other clusters’ paths under secret/data/ocp/<other-cluster>/....

The helper script ops-workspace/scripts/vault-eso-wire.sh is idempotent and creates both the auth mount and the role/policy on each cluster’s onboarding.

The CA bundle

caBundle is the most operationally annoying field. It must be:

• Base64-encoded (single line, no PEM headers stripped — the whole PEM block base64-encoded).
• The CA that signed the Vault server cert, not Vault’s leaf cert.

The lab Vault VM uses a long-lived self-signed CA; the bundle is committed (it’s a public-key cert, not a secret) at:

clusters/<cluster>/secrets/eso/clustersecretstore-vault.yaml#spec.provider.vault.caBundle

When Vault’s CA is rotated:

1. Generate the new bundle (base64 -w0 vault-ca.pem).
2. Update both clusters’ CSS YAMLs.
3. Push to GitOps; Argo reconciles each cluster within ~3 minutes.
4. Existing in-flight Vault tokens stay valid for their TTL; new auth requests use the new bundle automatically.

Smoke-test path

ESO ships a “smoke” pattern in the GitOps layout that proves CSS → ExternalSecret → Secret end-to-end:

# clusters/<cluster>/secrets/eso/smoke/externalsecret-smoke.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: eso-smoke
  namespace: external-secrets
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-platform
  target:
    name: eso-smoke
  data:
    - secretKey: hello
      remoteRef:
        key: ocp/<cluster>/eso-smoke
        property: hello

Vault data:

vault kv put secret/ocp/<cluster>/eso-smoke hello=world

Verification:

oc -n external-secrets get externalsecret eso-smoke \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
# True

oc -n external-secrets get secret eso-smoke \
  -o jsonpath='{.data.hello}' | base64 -d
# world

If the smoke value comes back, CSS auth, NetworkPolicy egress, and TokenReview are all working. From here, every other platform ExternalSecret is a permutation of path + secretKey + template.

Failure modes

References

• connection-details/vault-app-secrets.md (platform vs tenant path convention).
• connection-details/platform-admin-handoff.md (Vault wiring, helper scripts).
• ADRs: 0019 (image supply), 0024 (OpenShift-only platform gitops boundary).