app-registry pull Secret via ClusterExternalSecret

One ClusterExternalSecret materialises an app-registry-pull dockerconfigjson Secret into every namespace labelled apps.platform/tenant=true, instead of mutating the cluster-wide pull-secret.

The cluster does not patch openshift-config/pull-secret with the Nexus app-registry credential. Instead, a single ClusterExternalSecret (ESO) watches a label and fans a namespace-scoped Secret/app-registry-pull of type kubernetes.io/dockerconfigjson into every tenant namespace that opts in. This page is the manifest, the Vault path it reads, the label that triggers the fan-out, and the failure modes.

Per DEV-OCP-0.2 (#172). Paired with the tenant-template namespace, which ships the label out of the box.

Why not patch the cluster-wide pull-secret

OpenShift’s openshift-config/pull-secret, referenced by image.config.openshift.io/cluster, is the cluster-level auth registry list. Mutating it from GitOps means one of two bad options:

The pull secret is also rotated by separate concerns (cluster install, marketplace re-auth, registry CA changes). Sharing one Secret across “platform infrastructure” and “app-tier pull creds” couples two rotation cadences that should stay independent.

The alternative — a ClusterExternalSecret that fans a per-namespace Secret — keeps the cluster pull-secret immutable from GitOps, gives tenants opt-in semantics with a single label, and lets the app-registry credential rotate without touching the cluster-wide one.

The fan-out label

apps.platform/tenant: "true"

Every tenant namespace ships this label via the tenant template’s namespace.yaml. The label is load-bearing: forget it, the namespace gets no pull secret, the first Deployment fails with ImagePullBackOff.

The label is also a useful query target across the cluster:

oc get ns -l apps.platform/tenant=true

The Vault path

Single platform-owned path; tenants never see it.

secret/ocp/<cluster>/registries/app-registry-pull
  └── dockerconfigjson    (string; the full dockerconfigjson JSON blob)

Spoke value for spoke-dc-v6: secret/ocp/spoke-dc-v6/registries/app-registry-pull.

The value is the Nexus app-registry bot credential (platform-bot or similar), seeded at install by the platform admin via scripts/seed-app-registry-pull-secret.sh. The script enforces the canonical hostname (app-registry.apps.sub.comptech-lab.com, not app-registry.sub.comptech-lab.com — see failure modes).

Shape of the stored value:

{
  "auths": {
    "app-registry.apps.sub.comptech-lab.com": {
      "auth": "<base64(user:password)>",
      "email": "platform-bot@comptech-lab.local"
    }
  }
}

Until the path is populated, the ClusterExternalSecret reports Ready=False — desired state is still correct, it just cannot resolve the Vault data yet.

The manifest

Lives at platform-gitops/clusters/spoke-dc-v6/secrets/app-registry-pull/clusterexternalsecret.yaml:

apiVersion: external-secrets.io/v1
kind: ClusterExternalSecret
metadata:
  name: app-registry-pull
  annotations:
    argocd.argoproj.io/sync-wave: "45"
spec:
  externalSecretName: app-registry-pull
  refreshTime: 1h
  namespaceSelectors:
    - matchLabels:
        apps.platform/tenant: "true"
  externalSecretSpec:
    refreshInterval: 1h
    secretStoreRef:
      kind: ClusterSecretStore
      name: vault-platform
    target:
      name: app-registry-pull
      creationPolicy: Owner
      template:
        type: kubernetes.io/dockerconfigjson
        data:
          .dockerconfigjson: "{{ .dockerconfigjson }}"
    data:
      - secretKey: dockerconfigjson
        remoteRef:
          key: ocp/spoke-dc-v6/registries/app-registry-pull
          property: dockerconfigjson

Read top to bottom:

Wiring an existing namespace in

A namespace created before the apps.platform/tenant=true convention can be retrofitted with one command:

oc label ns apps-platform-liberty-hello-dev apps.platform/tenant=true
# Within ~60s ESO creates Secret/app-registry-pull in that namespace.

oc -n apps-platform-liberty-hello-dev get secret app-registry-pull \
  -o jsonpath='{.type}{"\n"}'
# Expected: kubernetes.io/dockerconfigjson

To use the secret, Pods either reference it on their own SA or the workload sets spec.imagePullSecrets. The tenant template ships a serviceaccount-default-pull-patch.yaml that patches the namespace’s default SA with imagePullSecrets: [{name: app-registry-pull}], so workloads using the default SA pull without per-workload config.

For workloads using a non-default SA (a common pattern for OperatorHub-installed apps), the SA needs to be patched explicitly:

oc -n <ns> patch sa <sa-name> \
  -p '{"imagePullSecrets":[{"name":"app-registry-pull"}]}'

RBAC dependency — already in place

ESO needs cluster-wide read on external-secrets.io/clusterexternalsecrets and write on secrets in every selected namespace. Both are covered by the spoke’s argocd-platform-extensions ClusterRole (covering 16 API groups including core secrets cluster-wide), shipped at clusters/spoke-dc-v6/platform/argocd-extensions/clusterrole.yaml.

A new tenant namespace does not need any additional RBAC — the platform RBAC is cluster-wide and selects namespaces by label.

Rotation

To rotate the bot credential:

# 1. Re-seed Vault with the new credential.
vault kv put secret/ocp/spoke-dc-v6/registries/app-registry-pull \
  dockerconfigjson="$(cat new-dockerconfig.json)"

# 2. Force ESO to re-pull immediately (otherwise wait < 1h).
oc annotate clusterexternalsecret app-registry-pull \
  force-sync=$(date +%s) --overwrite

# 3. Restart Pods that still hold the old credential (imagePullSecrets is admission-time).
oc -l apps.platform/tenant=true get ns -o name \
  | xargs -I {} oc -n {} rollout restart deployment --all 2>/dev/null

The third step is the common gotcha — imagePullSecrets is consumed at Pod admission time, so running Pods keep the old credential until restart. A rotation that doesn’t include a rollout will appear successful for Argo but fail the next time a Pod scales out or restarts.

Failure modes

Validation

# Cluster-wide health.
oc get clusterexternalsecret app-registry-pull \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
# Expected: True

# Count of namespaces that should have the Secret.
oc get ns -l apps.platform/tenant=true -o name | wc -l

# Count of namespaces that actually have it.
for ns in $(oc get ns -l apps.platform/tenant=true -o jsonpath='{.items[*].metadata.name}'); do
  oc -n "$ns" get secret app-registry-pull --no-headers 2>/dev/null \
    | awk -v n="$ns" '{print n, $0}'
done | wc -l

The two counts should match. If they don’t, inspect the ClusterExternalSecret status for the missing namespace under .status.failedNamespaces[].

What lives where

References

• DEV-OCP-0.2 (#172) — the ClusterExternalSecret/app-registry-pull design.
• ESO architecture
• Tenant scaffolding template — the namespace that ships the load-bearing label.
• ESO API: external-secrets.io/v1/ClusterExternalSecret.