Per-Tenant SecretStore: the vault-apps Pattern

How each apps-<division>-<app> namespace gets its own namespace-scoped SecretStore (vault-apps), bound to a per-division Vault role and policy. Onboarding flow, tenant template, ExternalSecret usage, and the cross-division isolation guarantees.

This is the tenant half of the secret-store split. The platform CSS (vault-platform) is for operators and platform namespaces; this page covers the namespace-scoped SecretStore vault-apps that lives in every tenant namespace and is the only way an app team’s ExternalSecret can talk to Vault.

What it is

SecretStore is the namespace-scoped peer of ClusterSecretStore. Same provider shape (Vault, kubernetes, etc.), but its visibility is constrained to one namespace, and a separate Vault role and policy back it. Three rules govern the tenant flavor:

1. Always SecretStore, never ClusterSecretStore. A tenant CSS would let any namespace read any tenant’s data; that’s a non-starter.
2. One per tenant namespace. Created by the tenant template (tenants/_template/secretstore-vault-apps.yaml) when the tenant overlay is materialized.
3. Bound to a per-division Vault role. The role’s namespace-glob is apps-<division>-*, so cross-division reads are refused by Vault, not by Kubernetes RBAC.

The Vault side (per division)

For division <division>, the Vault side is:

Policy:

path "secret/data/apps/<division>/*" {
  capabilities = ["read"]
}
path "secret/metadata/apps/<division>/*" {
  capabilities = ["list", "read"]
}

Role:

{
  "bound_service_account_names":      ["app-eso"],
  "bound_service_account_namespaces": ["apps-<division>-*"],
  "token_policies":                   ["apps-<division>-read"],
  "token_ttl":                        "1h",
  "token_max_ttl":                    "4h",
  "audience":                         "vault"
}

Three things this role enforces:

• ServiceAccount name. Only an SA named app-eso in a tenant namespace can authenticate. The tenant template creates this SA; no other SA in the namespace can read Vault.
• Namespace glob. The token will only issue if the SA’s namespace matches apps-<division>-*. Other divisions’ namespaces (or platform namespaces) cannot mint a token for this role.
• Policy. Once minted, the token can only read secret/data/apps/<division>/* — even within the same division, the role is read-only and path-pinned.

Onboarding a new division is one script (ops-workspace/scripts/vault-apps-onboard.sh):

scripts/vault-apps-onboard.sh <division> <cluster>
# e.g.
scripts/vault-apps-onboard.sh platform spoke-dc-v6

The script is idempotent — re-running rewrites the policy and role with the canonical body. There is no per-tenant secret material created at this step; only the policy and role.

The Kubernetes side (per tenant namespace)

The tenant template includes the manifest the tenant overlay must copy. The template lives at clusters/<cluster>/tenants/_template/secretstore-vault-apps.yaml:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-apps
  namespace: apps-<DIVISION>-<APP>
spec:
  provider:
    vault:
      server: https://vault.sub.comptech-lab.com:8200
      path: secret
      version: v2
      caBundle: <base64 vault CA — same as ClusterSecretStore vault-platform>
      auth:
        kubernetes:
          mountPath: kubernetes-<CLUSTER>
          role: apps-<CLUSTER>-<DIVISION>
          serviceAccountRef:
            name: app-eso
            audiences:
              - vault

The tokens to substitute on copy:

The CA bundle is the same string as in the ClusterSecretStore vault-platform. Copy it verbatim — there is one Vault CA, used by both platform and tenant paths.

The tenant overlay: full minimal shape

A tenant directory under clusters/<cluster>/tenants/apps-<division>-<app>/ contains, at minimum:

kustomization.yaml
namespace.yaml                  # Namespace + labels
serviceaccount-app-eso.yaml     # the app-eso SA
secretstore-vault-apps.yaml     # the SecretStore (this page)
networkpolicies/                # default-deny set (see §11)
resourcequota.yaml              # quotas (DEV-OCP-4.1)
limitrange.yaml                 # request/limit defaults (DEV-OCP-4.1)

The app-eso SA is the principal Vault sees:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-eso
  namespace: apps-<division>-<app>

No imagePullSecrets, no role bindings — the SA’s only purpose is to mint a projected JWT for Vault TokenReview. App workloads use other SAs.

How an ExternalSecret uses it

Once the SecretStore vault-apps is Ready=True, any ExternalSecret in that namespace can pull from Vault:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-password
  namespace: apps-platform-checkout
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore        # not ClusterSecretStore
    name: vault-apps
  target:
    name: db-password
    template:
      type: Opaque
  data:
    - secretKey: password
      remoteRef:
        key: apps/platform/checkout/prod/db
        property: password

The remoteRef.key is the Vault path relative to the kv-v2 mount (secret/data/ is added by ESO). The property is the field inside the kv-v2 entry.

The cross-division isolation guarantee

Why this matters: an attacker who compromises a tenant’s pod and reads the app-eso SA token cannot read any other tenant’s data, or the platform’s data, because:

1. The Vault role for division A is apps-<cluster>-A. Its bound_service_account_namespaces is apps-A-*, so a token presented from namespace apps-B-checkout is refused at TokenReview.
2. Even if the namespace glob were bypassed, the policy apps-A-read only grants read on secret/data/apps/A/*.
3. The platform path secret/data/ocp/<cluster>/* is not granted by any tenant policy, so platform secrets (pull tokens, RHACS bundle) are unreachable from tenant code.

Layered defense: Kubernetes RBAC enforces that tenant pods can only read Secrets in their own namespace; Vault enforces that the tenant SA can only mint a token with read on its own division’s subtree. Either layer alone would block cross-tenant reads; both together close the gap if one layer is misconfigured.

Onboarding flow for a new tenant

The end-to-end onboarding (assuming the division has already been onboarded via vault-apps-onboard.sh):

1. Create the tenant overlay under clusters/<cluster>/tenants/apps-<division>-<app>/ (the W1.C template scaffold).
2. Drop in secretstore-vault-apps.yaml from _template/, substitute the three tokens.
3. Drop in serviceaccount-app-eso.yaml so the SA exists before the SecretStore reconciles.
4. Push and merge via platform-gitops. Argo syncs within ~3 minutes.
5. Server-side seed the secret in Vault (not via GitOps):

   vault kv put secret/apps/<division>/<app>/<env>/<key> field=value

6. App team adds the ExternalSecret to their app repo overlay; references vault-apps.

Steps 1–4 are platform; step 5 is platform (Vault is a privileged system); step 6 is the tenant team in their own GitLab project.

Failure modes

Why no ClusterExternalSecret for tenants

ClusterExternalSecret is a CR that fans an ExternalSecret into many namespaces selected by label. It’s perfect for platform-wide fan-out (used for app-registry-pull, see §6 of the platform sidebar), but it’s wrong for tenants because:

• It references a ClusterSecretStore — bypassing the per-tenant isolation guarantee.
• Targets multiple namespaces, which conflicts with “this tenant owns this secret.”

Tenants always use per-namespace ExternalSecret against their per-namespace SecretStore. The only ClusterExternalSecret in the system is platform-owned.

References

• connection-details/vault-app-secrets.md — the full onboarding convention (DEV-OCP-0.4 / #174).
• tenants/_template/secretstore-vault-apps.yaml in platform-gitops.
• scripts/vault-apps-onboard.sh in ops-workspace/scripts/.
• ADRs: 0014 (developer readiness), 0023 (federated GitLab group/repo ownership).