ESO Secret bridge and cluster-wide pull Secret

The per-tenant SecretStore that bridges ESO to Vault, and the ClusterExternalSecret that fans an app-registry pull Secret into every namespace labelled apps.platform/tenant=true.

Two secret-delivery mechanisms live side by side in a tenant namespace:

The two are intentionally separate: tenants control their app secrets; the platform controls the image-registry credential. Tenants never see the pull secret’s value — they only opt-in their namespace by labelling it.

Per-tenant SecretStore — the ESO bridge

Covered in detail at 02 — Vault path and bound role. One-paragraph summary here:

Each tenant namespace ships a SecretStore/vault-apps (NOT ClusterSecretStore) that points the namespace’s app-eso ServiceAccount at the Vault role apps-<cluster>-<division>. That role has policy apps-<division>-read which can read secret/data/apps/<division>/* only. Tenants then write ExternalSecret resources in their app overlay (see the app-repo overlay contract) that materialise Kubernetes Secret objects from Vault paths.

The platform-provided template at platform-gitops/clusters/<cluster>/tenants/_template/secretstore-vault-apps.yaml is the only blessed copy.

Cluster-wide app-registry pull Secret — the design

Per issue #172 (DEV-OCP-0.2).

Why not patch the cluster-wide pull-secret?

OpenShift’s cluster-wide pull secret (openshift-config/pull-secret, referenced by image.config.openshift.io/cluster) is intentionally not mutated for app-registry credentials. Merging an additional auth entry into that Secret would either require a full replace (risky — drops every other registry the cluster pulls from) or out-of-band tooling that fights desired-state GitOps.

Instead, the spoke uses a ClusterExternalSecret (ESO) that fans a namespace-scoped Secret/app-registry-pull of type kubernetes.io/dockerconfigjson into every namespace labelled:

apps.platform/tenant=true

Workloads opt in per-namespace by adding app-registry-pull to the default ServiceAccount’s imagePullSecrets (or per-Pod spec.imagePullSecrets).

This keeps the global pull secret immutable from GitOps and lets tenant namespaces enable app-registry pulls with a single label + one SA patch.

Where it lives

ClusterExternalSecret shape

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: app-registry-pull
spec:
  externalSecretName: app-registry-pull
  namespaceSelectors:
    - matchLabels:
        apps.platform/tenant: "true"
  refreshTime: 1h
  externalSecretSpec:
    secretStoreRef:
      kind: ClusterSecretStore
      name: vault-platform                        # platform-scoped store, not the per-tenant one
    target:
      name: app-registry-pull
      template:
        type: kubernetes.io/dockerconfigjson
        data:
          .dockerconfigjson: "{{ .dockerconfigjson }}"
    data:
      - secretKey: dockerconfigjson
        remoteRef:
          key: ocp/spoke-dc-v6/registries/app-registry-pull
          property: dockerconfigjson

Notes:

• Backed by ClusterSecretStore/vault-platform, not the tenant’s SecretStore. The credential is platform-owned.
• The Vault value is stored as a full dockerconfigjson JSON blob; the ESO template wraps it in the .dockerconfigjson key Kubernetes expects.
• refreshTime: 1h — fast enough to catch a credential rotation within a sync window, slow enough not to hammer Vault.

The Vault value shape

{
  "auths": {
    "app-registry.apps.sub.comptech-lab.com": {
      "auth": "<base64(user:password)>",
      "email": "platform-bot@comptech-lab.local"
    }
  }
}

The credential is the Nexus app-registry bot account (platform-bot or similar), seeded at install time by the platform admin. Tenants never see this credential.

Until this path is populated server-side, the ClusterExternalSecret reports Ready=False — the desired state is still correct, it just cannot resolve the Vault data yet.

Tenant opt-in — the three commands

The tenant onboarding template ships these as YAML so the operator never types them by hand. The equivalent imperative form:

# 1. Label the tenant namespace.
oc label ns apps-platform-liberty-hello-dev apps.platform/tenant=true

# ESO will create Secret/app-registry-pull in that namespace within the refresh window
# (≤ 1 hour; usually ≤ 1 min after ESO sees the namespace).

# 2. Confirm it materialised.
oc -n apps-platform-liberty-hello-dev get secret app-registry-pull \
  -o jsonpath='{.type}{"\n"}'
# Expected: kubernetes.io/dockerconfigjson

# 3. Attach the pull secret to the namespace default ServiceAccount.
oc -n apps-platform-liberty-hello-dev patch sa default \
  -p '{"imagePullSecrets":[{"name":"app-registry-pull"}]}'

For workloads using a non-default ServiceAccount, attach the pull secret to that SA instead — or set imagePullSecrets directly on the workload Pod spec (less preferred; couples the workload to the secret name).

When to also use the Quay pull Secret

Apps whose images live in the in-cluster Quay (Path B) need a Quay pullSecret too. The pattern is identical: a second ClusterExternalSecret/quay-pull (when Quay is up) fans a Secret/quay-pull into the same tenant namespaces, and the operator patches the SA’s imagePullSecrets to include both:

oc -n apps-platform-quay-only-sample-dev patch sa default \
  -p '{"imagePullSecrets":[{"name":"app-registry-pull"},{"name":"quay-pull"}]}'

The Path B Quay-only sample at 05 — Path-B end-to-end walks the operator through this.

Per-tenant Quay robot token (for CI push, not runtime pull)

A different Quay credential is needed at CI time for the build pipeline to push images. This one is per-tenant, lives in the tenant’s Vault path under ci/, and materialises in openshift-pipelines (not the tenant namespace):

Vault path: secret/apps/<division>/<app>/ci/quay-robot
Materialised Secret: openshift-pipelines/quay-robot-team-<team>
Read by: the shared Tekton Task `push-image-quay`

The ClusterExternalSecret for the cluster-wide pull Secret has no role here — runtime pull and CI push are different concerns with different credentials. See 09 — Per-tenant Quay robot token for the full pattern.

Validation

After onboarding, confirm the three Secrets exist:

NS=apps-platform-liberty-hello-dev

# 1. Cluster-wide pull Secret materialised.
oc -n $NS get secret app-registry-pull \
  -o jsonpath='{.type}{"\n"}'
# Expected: kubernetes.io/dockerconfigjson

# 2. Default SA references it.
oc -n $NS get sa default -o jsonpath='{.imagePullSecrets}{"\n"}'
# Expected: [{"name":"app-registry-pull"}]

# 3. Tenant SecretStore exists.
oc -n $NS get secretstore vault-apps \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
# Expected: True

# 4. ClusterExternalSecret reports Ready cluster-wide.
oc get clusterexternalsecret app-registry-pull \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
# Expected: True

Failure modes

References

• opp-full-plat/connection-details/app-registry-pullsecret.md (issue #172, DEV-OCP-0.2) — full design.
• opp-full-plat/connection-details/vault-app-secrets.md (issue #174, DEV-OCP-0.4) — per-tenant SecretStore.
• Platform memory project_eso_egress_to_vault.md — NetworkPolicy gotcha.
• ESO ClusterExternalSecret API — external-secrets.io/v1beta1.
• Kubernetes imagePullSecrets semantics on ServiceAccount vs Pod.