Credential Custody Map

Where every credential class lives — Vault, ESO-materialized Secret, htpasswd, local-only file, or operator-managed — owner, rotation, and reference doc.

The lab’s credentials live in a small number of well-defined custody planes. This page is the map. For each credential class it lists the custodian (where the value lives), the owner, the rotation policy, and the canonical operator-doc reference. No credential values appear on this page.

Custody planes

Plane	What lives there
Vault (KV-v2)	Long-lived secrets consumed by clusters via ESO, plus a few local-automation tokens.
ESO-materialized `Secret`	Per-namespace Kubernetes Secrets created by ESO from Vault. Treated as ephemeral copies.
OBC-generated `Secret`	NooBaa OBC creates a Secret in the operand namespace. Bridged into operand-shaped Secret via ESO.
htpasswd (in-cluster)	OpenShift identity-provider bootstrap and a few operator-bootstrap credentials (RHACS Central admin).
Local-only file	Workstation files under `opp-full-plat/secrets/` (local-only, contents not enumerated here) for bootstrap credentials that pre-date Vault.
Operator-managed	Generated by an operator (e.g., NooBaa OBC, cert-manager, ACM). Not human-edited.

Credential classes

Credential class	Custodian	Owner	Rotation policy	Reference
Vault root token + unseal keys	Local-only (out-of-band) on operator workstation	platform-admin	Manual; on operator change	`plans/disconnected-rebuild/environments/dc-lab/vault-oss-vm-plan.md`
Vault snapshot token	Scoped Vault token (not root)	platform-admin	Per Vault policy	`vault-oss-vm-plan.md`
OpenShift kubeadmin password (hub)	Local-only file at the hub kubeadmin password file (local-only)	platform-admin	Manual rotate after install steady state	`connection-details/platform-admin-handoff.md`
OpenShift kubeadmin password (spoke)	Local-only file at the spoke kubeadmin password file (local-only)	platform-admin	Manual rotate after install steady state	`connection-details/platform-admin-handoff.md`
Nexus `admin` password	Local-only file at the Nexus admin password file (local-only)	platform-admin	Manual; do not share	`connection-details/nexus.md`
Nexus `jenkinsbot` service account	Local-only file at the Nexus jenkinsbot password file (local-only)	platform-admin (Jenkins consumer)	Manual; rotate on suspected leak	`connection-details/nexus.md`
Nexus OAuth2 cookie secret	Local-only file at the Nexus OAuth2 cookie-secret file (local-only)	platform-admin	On compromise	`connection-details/nexus.md`
MinIO CI evidence writer	Local-only file at `secrets/minio-developer-ci-evidence.env`	platform-admin	Manual	`connection-details/minio.md`
MinIO CI evidence reader	Local-only file at `secrets/minio-developer-ci-evidence-reader.env`	platform-admin	Manual	`connection-details/minio.md`
GitLab admin PAT	Vault `secret/platform/gitlab/admin-pat`	platform-admin	Per GitLab PAT policy	`plans/disconnected-rebuild/gitlab-bootstrap-design.md`
GitLab runner registration token (per class)	Vault `secret/platform/gitlab-runner/<class>/registration-token`	platform-admin	On runner rebuild	`plans/gitlab-runner-classes-and-tags.md`
GitLab MR creator PAT	Local-only file at the bootstrap GitLab PAT file (local-only)	platform-admin	Manual	`reference_platform_gitops_mr_path.md`
Cluster ESO Kubernetes-auth reviewer SA	In-cluster SA + Vault auth mount config	platform-admin	On Vault re-init	`vault-oss-vm-plan.md`
App tenant Vault paths (`secret/apps/<division>/<app>/...`)	Vault KV-v2	division team	Per app policy	`connection-details/vault-app-secrets.md`
App tenant ESO `SecretStore`	Namespace-scoped CR in tenant namespace	division team	GitOps-managed	`connection-details/vault-app-secrets.md`
Quay robot token (per team)	Vault `secret/apps/<division>/<app>/ci/quay-robot`	division team	Per team policy	`reference_quay_robot_token_convention.md`
Quay config bundle (in-cluster operand)	Vault `secret/ocp/spoke-dc-v6/quay/config-bundle`	platform-admin	On Quay redeploy	`project_obc_to_operand_secret_bridge.md`
App registry pull secret	Vault `secret/ocp/spoke-dc-v6/registries/app-registry-pull`	platform-admin	On Nexus credential rotate	`connection-details/app-registry-pullsecret.md`
OADP S3 credentials	OADP `DataProtectionApplication` Secret in `openshift-adp`	platform-admin	On MinIO key rotate	OADP install runbook
NooBaa OBC creds (Loki / Tempo / Quay)	NooBaa-generated `Secret` + `ConfigMap` in operand namespace	operator-managed	OBC lifecycle	`project_obc_to_operand_secret_bridge.md`
RHACS Central admin (htpasswd)	In-cluster Secret in `stackrox` namespace	platform-admin	On Central reinstall	`reference_rhacs_init_bundle_via_api.md`
RHACS init bundle (kubectlBundle)	Vault for ESO delivery; flattened from Central API	platform-admin	On bundle regen	`reference_rhacs_init_bundle_via_api.md`
Let’s Encrypt wildcard private keys (`.apps.sub.`, `.mon.sub.`)	On HAProxy VM under `/etc/haproxy/certs/`	platform-admin	LE renewal (90d)	`reference_haproxy_vm.md`
ACME `dns_pdns` credentials	On `pdns` VM under `acme.sh` user	platform-admin	On PowerDNS API key rotate	`reference_pdns_vm.md`
OIDC client secrets (WSO2 / Nexus / GitLab / Grafana)	Local-only file under `secrets/` (e.g., the Nexus OIDC client config (local-only))	platform-admin	On IdP rebuild	per-service connection-detail doc
SSH custody for platform VMs	Local-only on operator workstation	platform-admin	Per personal key rotation policy	`reference_lab_infrastructure.md`

Custody plane: Vault

Path conventions and policies are detailed in Vault Paths. Highlights:

secret/ocp/<cluster>/* -> per-cluster platform secrets read via ESO.
secret/platform/* -> lab-automation tokens (GitLab, MinIO HMAC, runners).
secret/apps/<division>/<app>/<env>/* -> tenant secrets read by tenant SecretStore (vault-apps) into the tenant namespace.

Custody plane: ESO-materialized `Secret`

ESO consumes Vault paths and writes namespace-scoped Kubernetes Secrets. These Secrets are treated as ephemeral: GitOps must not manage them directly, and they must not be backed up. The ExternalSecret manifest is the source of truth — re-creating the ExternalSecret re-creates the Secret.

Custody plane: htpasswd

Used only where an operator’s bootstrap design requires it:

OpenShift cluster admin identity provider (post-kubeadmin handover) — managed by the cluster install GitOps.
RHACS Central admin — generated at install, captured via the Central API for downstream init-bundle generation (per reference_rhacs_init_bundle_via_api.md).

Custody plane: local-only files

Files under opp-full-plat/secrets/ (local-only, contents not enumerated here) are mode-restricted, git-ignored, and never copied off the operator workstation. They cover the gap between “cluster + Vault are up” and “Vault is the system of record”. The directory is the canonical custody location for:

kubeadmin passwords (hub + spoke).
Nexus admin + service-account passwords + OIDC client.
MinIO CI evidence keys (writer + reader).
GitLab MR-creator PAT.
Vault unseal/root material (out-of-band copy).

The .prev and .stale-pre-* siblings are previous custody — not for use unless rolling back.

Rotation policy notes

Class	Cadence	Trigger
LE wildcard certs	Every renewal (~60-90 days)	ACME timer on HAProxy VM.
Nexus service accounts	On suspected leak	Manual; re-issue + restart Jenkins job creds.
Vault root token	Never used for automation	Use a scoped role / policy instead.
App tenant Vault paths	Per division team	Onboarding script + tenant policy.
GitLab admin PAT	Per GitLab policy	Stored in Vault `secret/platform/gitlab/admin-pat`.
OpenShift kubeadmin	After installing real IdP	Then disable kubeadmin per OCP guidance.

Failure modes

Symptom	Root cause	Fix
ESO `Status: Failed`, “permission denied” on Vault path	Tenant SA not in role’s namespace glob	Re-run onboarding with correct division name.
ESO operand Secret missing keys	OBC -> operand shape mismatch	Apply the OBC -> operand ExternalSecret bridge.
Jenkins push fails 403 on `app-registry.*`	`jenkinsbot` credentials rotated but Jenkins not updated	Update Jenkins credential `nexus-jenkinsbot` and rerun.
Cert expiry alert from HAProxy	`acme.sh` renewal failed	Check PowerDNS API key + acme.sh log on `pdns` VM.

Internal only

All concrete credential values are kept off this site. Local custody paths are listed by filename above; the values themselves are in opp-full-plat/secrets/ or in Vault.

Last regenerated from connection-details/platform-admin-handoff.md, connection-details/nexus.md, connection-details/minio.md, connection-details/vault-app-secrets.md, reference_lab_infrastructure.md, reference_quay_robot_token_convention.md, reference_rhacs_init_bundle_via_api.md, reference_haproxy_vm.md.