Secret engines and path conventions

Vault OSS has no namespaces, so strict path prefixes do the segmentation. The lab's path tree under secret/ — ocp/platform, ocp/<cluster>, apps/<division>/<app>/<env> — and the KV-v2 read/write conventions.

Vault OSS has no Enterprise namespaces. The lab compensates by using strict path prefixes under a single KV-v2 mount. This page documents the prefix tree, the KV-v2 semantics, the read/write paths, and the rules for adding new paths.

The single KV-v2 mount

Mount	Type	Path
`secret/`	KV-v2	enabled by default during init

All lab secrets live under secret/. No kv-pki/, no kv-app/, no per-app mounts. One mount, many prefixes, one mental model.

KV-v2 supports versioning, soft delete, and CAS (check-and-set). The lab enables versioning (default) so accidental overwrites are recoverable.

Path tree

secret/
  ocp/
    platform/                       # shared across clusters
      <secret-name>
    hub-dc-v6/
      <secret-name>
    spoke-dc-v6/
      <secret-name>
  apps/
    <division>/                     # e.g., platform, payments, retail
      <app>/                        # e.g., open-liberty-readiness-probe
        dev/
          <key>
        stage/
          <key>
        prod/
          <key>

Examples:

Logical secret	Path
Nexus admin password (platform shared)	`secret/ocp/platform/nexus-admin`
Hub-cluster kubeadmin (cluster-local)	`secret/ocp/hub-dc-v6/kubeadmin`
ESO smoke probe (tenant)	`secret/apps/platform/eso-smoke/dev/hello`
Payments DB password (tenant, prod)	`secret/apps/payments/checkout-api/prod/db.password`
RHACS init bundle (platform)	`secret/ocp/platform/rhacs/init-bundle`
Quay robot pull-secret (tenant)	`secret/apps/<division>/<app>/ci/quay-robot`

KV-v2 read/write paths (the gotcha)

KV-v2 has two paths per secret:

Data path: secret/data/<prefix>/<key> — reads return the latest version of the secret.
Metadata path: secret/metadata/<prefix>/<key> — lists, version history, soft-delete operations.

When you write or read at the CLI, you use the logical path:

vault kv put secret/apps/platform/eso-smoke/dev hello=world
vault kv get secret/apps/platform/eso-smoke/dev
vault kv list secret/apps/platform/

When you write policy ACLs, you write against the real HTTP API paths:

path "secret/data/apps/<division>/*" {
  capabilities = ["read"]
}
path "secret/metadata/apps/<division>/*" {
  capabilities = ["list", "read"]
}

The /data/ vs /metadata/ distinction trips up most newcomers. Vault’s policy grants are underneath the data/metadata indirection; the CLI hides it. ESO’s remoteRef.key looks like the logical path too:

data:
  - secretKey: foo
    remoteRef:
      key: apps/platform/eso-smoke/dev/hello
      property: hello

(No secret/data/ prefix — ESO infers it from the SecretStore config.)

Why this prefix shape

Choice	Rationale
Single `secret/` mount	Vault OSS limit: no namespaces; one KV-v2 mount keeps the model simple.
`ocp/` vs `apps/` top split	Platform secrets vs tenant secrets are different audiences. Platform ESO reads `ocp/`; tenant ESO reads `apps/`. The policies enforce the boundary.
`ocp/<cluster>` not `ocp/clusters/<cluster>`	Saves one level; the cluster is the meaningful axis at platform scope.
`apps/<division>/<app>/<env>/<key>`	Four levels: division → app → environment → key. Each level is something a different role would scope on. Division → policy. App → app’s secret namespace. Env → promotion gating. Key → the actual leaf.
`<env>` inside `apps/<division>/<app>/`	Per `vault-app-secrets.md`. Apps almost always have dev/stage/prod separation; bake it in.

Adding a new path

Three cases:

Adding a platform secret

vault kv put secret/ocp/platform/<name> field1=value1 field2=value2

The platform ESO policy (ocp-<cluster>-eso) already covers secret/data/ocp/platform/* and secret/metadata/ocp/platform/* — no policy edit needed.

Adding a tenant secret for an existing division

vault kv put secret/apps/platform/<app>/<env>/<key> <field>=<value>

If the division already has its apps-<division>-read policy and apps-<cluster>-<division> role wired, no policy edit needed. The tenant ESO consumer in the matching namespace reads the new path on next sync cycle.

Onboarding a new division

Per connection-details/vault-app-secrets.md:

Pick a division name (lowercase, no underscores, e.g., payments).
Run the onboarding script from the operator workstation:
```
/home/ze/ops-workspace/scripts/vault-apps-onboard.sh <division> <cluster>
```
This creates/updates:
- Policy apps-<division>-read
- Role apps-<cluster>-<division> under auth/kubernetes-<cluster>/
Create the tenant namespace via GitOps with a SecretStore named vault-apps pointing at the new role.

Seed the secret in Vault:

vault kv put secret/apps/<division>/<app>/<env>/<key> <field>=<value>

Reference it from the app’s ExternalSecret.

The onboarding script is idempotent — running it twice produces the same end state.

What does not go in `secret/`

Secret type	Where it lives instead
TLS private keys for Vault itself	`/etc/vault.d/tls/vault.key` on each voter
Lab CA root + intermediate keys	Offline custody (operator workstation, encrypted)
Vault root / unseal shares	Offline custody, separate from Vault VMs
Vault snapshot scoped token	Local-only `/etc/vault.d/snapshot-token` on the snapshot-running voter
MinIO root credentials	Local-only `secrets/minio/root.env` on operator workstation
MinIO scoped IAM user credentials	Mostly in Vault under `secret/ocp/platform/minio/*` for shared services; in-cluster operands use OBC bridge
Long-term keys for operator GitHub PAT / GitLab PAT	Local-only `secrets/` directory on operator workstation; not yet migrated to Vault

The principle: anything that’s needed before Vault is up cannot live in Vault. That’s why the lab CA, Vault’s own TLS, Vault’s own unseal shares all live outside.

Reads from KV-v2

# Read the latest version
vault kv get secret/apps/platform/eso-smoke/dev

# Read a specific version
vault kv get -version=3 secret/apps/platform/eso-smoke/dev

# Get a single field
vault kv get -field=hello secret/apps/platform/eso-smoke/dev

# Show metadata only
vault kv metadata get secret/apps/platform/eso-smoke/dev

Soft delete and version retention

KV-v2 keeps the last N versions of each secret (default 10). Operations:

# Soft-delete the latest version (recoverable)
vault kv delete secret/apps/platform/eso-smoke/dev

# Undelete
vault kv undelete -versions=5 secret/apps/platform/eso-smoke/dev

# Hard-delete (destroys a specific version forever)
vault kv destroy -versions=5 secret/apps/platform/eso-smoke/dev

# Configure version retention on a path
vault kv metadata put -max-versions=20 secret/apps/platform/eso-smoke/dev

The lab leaves max-versions at the default (10). Critical platform secrets sometimes get bumped to 20.

Path naming style

Convention	Why
All lowercase	Vault paths are case-sensitive; lowercase keeps grep predictable
Hyphens for word separation	(`pgsql-readonly`, not `pgsql_readonly` or `pgsqlReadonly`)
No spaces, no Unicode	Plays badly with URLs and shell quoting
Singular, role-named keys	`db.password`, `api.key`, `tls.cert` — not `passwords` or `creds`
Lock to the `<division>/<app>/<env>/<key>` shape for tenants	Predictable for ESO `remoteRef.key` consumers

Failure modes

Symptom	Root cause	Fix	Prevention
ESO sync fails with “key not found” but the secret exists	`remoteRef.key` includes `secret/data/` prefix	Remove the `secret/data/` from the ESO `remoteRef.key`	Use logical path (no `data/`) in `remoteRef.key`
ACL denies access despite the policy looking right	Policy points at `secret/<prefix>/` instead of `secret/data/<prefix>/`	Update policy to use the `data/` HTTP API path	Always write KV-v2 policies against the `data/` and `metadata/` paths
`vault kv list` returns nothing for a path that has secrets	Policy doesn’t grant `list` on `secret/metadata/<prefix>`	Add `list` capability on the metadata path	Always grant `list` on metadata in any reader policy
Secret destroyed irrecoverably	Used `vault kv destroy` instead of `delete`	Restore from a Raft snapshot if recent enough	Use `delete` (recoverable), not `destroy`; reserve `destroy` for explicit cleanup
Wrong division can read another’s path	A tenant’s namespace name matched another division’s namespace glob (e.g., a `payments-*` role glob too permissive)	Tighten role glob; revoke the misissued tokens	Always include the division name in the namespace glob: `apps-<division>-*`

References

opp-full-plat/connection-details/vault-app-secrets.md
opp-full-plat/plans/disconnected-rebuild/environments/dc-lab/vault-oss-vm-plan.md
HashiCorp KV v2 docs: developer.hashicorp.com/vault/docs/secrets/kv/kv-v2