App secret namespacing

The per-division tenancy convention — one Vault policy per division, namespace-scoped tenant SecretStore (never ClusterSecretStore), and the onboarding flow for new divisions and new apps.

The lab’s per-division tenancy convention (formalized in connection-details/vault-app-secrets.md and DEV-OCP-0.4 / issue #174) is how application secrets stay segmented in a Vault OSS deployment that has no namespaces feature. This page is the full convention.

Goals (verbatim from the spec)

One Vault path subtree per division: secret/apps/<division>/<app>/<env>/*
Each division gets its own Vault policy + role; no shared apps-read policy.
ESO bindings are per-tenant SecretStore (namespace-scoped), never a ClusterSecretStore. Each tenant namespace runs its own ESO service account.
Onboarding is driven by scripts/vault-apps-onboard.sh (server-side) followed by a GitOps overlay that drops the tenant SecretStore into clusters/<cluster>/tenants/<tenant>/.

The four-axis identity

A tenant secret has four identity axes:

Axis	Example	Lives in
Division	`platform`	First path segment under `apps/`; Vault role name; Vault policy name; namespace prefix
App	`open-liberty-readiness-probe`	Second path segment; usually also the namespace’s app suffix
Environment	`dev` / `stage` / `prod`	Third path segment; matches the tenant’s dev/stage/prod separation
Key	`db.password`	Fourth path segment; the leaf secret name

The first axis (division) is the primary segmentation. The Vault policy enforces “this token can only read paths starting with apps/<division>/.” The role binding enforces “this token can only be obtained by a SA in a namespace matching apps-<division>-*.”

Vault role + policy naming

Per cluster, per division:

Role:    apps-<cluster>-<division>      e.g., apps-spoke-dc-v6-platform
Policy:  apps-<division>-read           e.g., apps-platform-read   (cluster-agnostic)
Path:    secret/apps/<division>/...     e.g., secret/apps/platform/...

The policy is shared across clusters (a single apps-platform-read is used by both hub and spoke). The role pins which cluster and which namespace glob can attach to that policy. So a platform division secret can be read by ESO in any cluster, but only by an app-eso SA in an apps-platform-* namespace.

The role binding

{
  "bound_service_account_names":      ["app-eso"],
  "bound_service_account_namespaces": ["apps-<division>-*"],
  "token_policies":                   ["apps-<division>-read"],
  "token_ttl":                        "1h",
  "token_max_ttl":                    "4h",
  "audience":                         "vault"
}

app-eso is the per-tenant ServiceAccount that the tenant template creates in each apps-<division>-<app> namespace.
Not the same SA as the platform ESO (external-secrets-operator-controller-manager). Platform and tenant ESO are two different identities reading two different parts of the path tree.
Namespace glob apps-<division>-* matches apps-platform-sample, apps-platform-readiness-probe, etc. Always include the division name in the glob — otherwise a tenant in a different division could mount that role.
TTLs match the platform ESO role (1h / 4h).

The policy

path "secret/data/apps/<division>/*" {
  capabilities = ["read"]
}
path "secret/metadata/apps/<division>/*" {
  capabilities = ["list", "read"]
}

That’s it. Read-only on the data path, list+read on metadata. No create, no update, no delete. Secret seeding happens server-side via the onboarding script, not through ESO.

Per-tenant `SecretStore` shape

A tenant namespace gets its own SecretStore (not ClusterSecretStore):

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-apps
  namespace: apps-<DIVISION>-<APP>
spec:
  provider:
    vault:
      server: https://vault.sub.comptech-lab.com:8200
      path: secret
      version: v2
      caBundle: <base64 lab CA>
      auth:
        kubernetes:
          mountPath: kubernetes-<CLUSTER>
          role: apps-<CLUSTER>-<DIVISION>
          serviceAccountRef:
            name: app-eso
            audiences:
              - vault

Key points:

kind: SecretStore, not ClusterSecretStore. Namespace-scoped.
server: https://vault.sub.comptech-lab.com:8200 — the DNS RR endpoint. Same value in every tenant store.
caBundle is the base64’d lab CA chain — the same value as in the platform clustersecretstore-vault.yaml. Copy verbatim, never invent.
mountPath: kubernetes-<CLUSTER> — picks the per-cluster K8s auth mount.
role: apps-<CLUSTER>-<DIVISION> — pins the role.
serviceAccountRef.name: app-eso — the per-tenant SA that already exists in the namespace.
audiences: [vault] — requests a JWT specifically for Vault.

A placeholder copy lives at clusters/spoke-dc-v6/tenants/_template/secretstore-vault-apps.yaml. The tenant template (W1.C) copies it into each new tenant directory and substitutes <DIVISION>, <APP>, and <CLUSTER>.

Onboarding a new division

From connection-details/vault-app-secrets.md:

Pick a division name. Lowercase, no underscores. Becomes part of:
- Namespace prefix (apps-<division>-*)
- Vault role (apps-<cluster>-<division>)
- Vault policy (apps-<division>-read)
- Path subtree (secret/apps/<division>/...)
Run the onboarding script (server-side, on the operator workstation):
```
/home/ze/ops-workspace/scripts/vault-apps-onboard.sh <division> <cluster>
# e.g., /home/ze/ops-workspace/scripts/vault-apps-onboard.sh platform spoke-dc-v6
```
Creates/updates:
- Policy apps-<division>-read
- Role apps-<cluster>-<division> under auth/kubernetes-<cluster>/
Idempotent — re-running rewrites the policy and role with the canonical body.
Create the tenant namespace via GitOps. The tenant template drops a directory under clusters/<cluster>/tenants/apps-<division>-<app>/. Copy the SecretStore from _template/secretstore-vault-apps.yaml, substituting <DIVISION>, <APP>, and <CLUSTER>.

Seed the secret in Vault (server-side, not via GitOps):

vault kv put secret/apps/<division>/<app>/<env>/<key> <field>=<value>

Reference it from the app’s ExternalSecret:

spec:
  secretStoreRef:
    kind: SecretStore
    name: vault-apps
  data:
    - secretKey: foo
      remoteRef:
        key: apps/<division>/<app>/<env>/<key>
        property: <field>

The ExternalSecret lives in GitOps; the secret data lives in Vault. Nothing about the secret value is ever in the planning repo or in Git.

Why per-tenant, not ClusterSecretStore

Two reasons:

Namespace scoping. Each tenant namespace owns its own store. It cannot read other divisions’ paths because:
- The role’s namespace glob refuses to issue a token outside apps-<division>-*.
- Even if it tried, the policy doesn’t grant read on other divisions’ paths.
Blast radius. A misconfigured ClusterSecretStore could silently expose paths cluster-wide. A tenant SecretStore is one namespace, one role, one policy. Easier to audit.

The platform ClusterSecretStore (clusters/<cluster>/secrets/eso/clustersecretstore-vault.yaml) still exists — that’s the platform ESO controller’s view, used for platform secrets (ocp/platform/*, ocp/<cluster>/*). It is never used for tenant secrets.

Cross-cluster considerations

A division’s apps in spoke-dc-v6 and hub-dc-v6 both read from secret/apps/<division>/... using the same policy. Each cluster has its own role under its own K8s auth mount, so:

A spoke-cluster JWT can only get the spoke-cluster role’s token.
A hub-cluster JWT can only get the hub-cluster role’s token.
Both roles attach the same policy (apps-<division>-read), reading the same Vault paths.

This means: secrets are shared across clusters per division, but JWT-issued credentials are per-cluster. Revoking a cluster’s reviewer JWT (auth mount config) instantly blocks that cluster’s apps from reading any division’s secret, without affecting the other cluster.

What this protects against

Risk	Mitigation
Tenant accidentally reads another tenant’s secret	Path prefix + namespace glob both have to align
Tenant compromise propagates across clusters	Each cluster has its own auth mount with its own reviewer JWT
Platform secrets leak into tenant ESO	Tenant policy doesn’t cover `ocp/*` paths at all
A new operator forgets to scope a new division	The onboarding script is the only blessed path; it sets the right policy and role

What it does NOT protect against

Vault root compromise. Root can read anything. Operator practice + offline custody is the mitigation.
Compromise of the per-tenant app-eso SA inside a tenant namespace. That SA legitimately can read the division’s secrets. The blast radius is bounded by the division’s path subtree.
Policy errors during division onboarding. The script is idempotent and reviewed; manual policy edits should be rare.

Failure modes

Symptom	Root cause	Fix	Prevention
Tenant ESO syncs nothing	`SecretStore` `serviceAccountRef.name` doesn’t match the SA actually deployed	Reconcile the SA + the store reference	Pair the SA + the store in the same tenant manifest
`403` from Vault during sync	Role’s namespace glob too narrow (or division name typo)	Re-run onboarding with the correct division name	Run a smoke `ExternalSecret` before any real apps land
ESO reports “no such role”	Tenant’s `mountPath` or `role` doesn’t match what Vault has	Verify with `vault read auth/kubernetes-<cluster>/role/apps-<cluster>-<division>`	Always run the onboarding script before applying the tenant `SecretStore`
Secret leaks via wrong-path commit	An operator wrote a value into the GitOps repo or planning repo	Rotate the secret immediately; clean the Git history if absolutely necessary	The ExternalSecret schema doesn’t include values; treat any value in Git as a finding

References

opp-full-plat/connection-details/vault-app-secrets.md
opp-full-plat/plans/disconnected-rebuild/environments/dc-lab/vault-oss-vm-plan.md
ESO docs: external-secrets.io/latest
HashiCorp Vault Kubernetes auth: developer.hashicorp.com/vault/docs/auth/kubernetes