Rotate secrets and tokens

The end-to-end rotation procedure for the platform's credential surface: Vault, Nexus, GitLab, GitHub, RHACS, Quay, robot pulls. Probe first, generate safely, rotate everywhere in one window.

This page is the playbook for rotating a credential anywhere on the platform’s surface — Vault tokens, Nexus per-bot passwords, GitLab PATs, GitHub PATs, RHACS Central admin / init-bundle, Quay robot pull secrets, MinIO access keys. The procedure is identical in shape across credential types; the per-credential details differ.

The rule that makes the procedure work: rotate on the live system first, then update Vault and the local mirror in the same change window. Skipping either side of that pair is the silent drift the secrets custody drift check is designed to catch.

When this task runs

• Quarterly cadence. Per ADR 0021-class custody policy, every long-lived credential rotates at least every 90 days.
• Offboarding event. When a named human or service identity leaves the team, every credential they touched rotates immediately.
• Drift-check failure. When secrets-custody-drift-check.md probes return 401, WRONGPASS, or similar, the credential is rotated as part of the recovery.
• Suspected leak. A credential in chat / issue body / commit message triggers immediate rotation.

What is in scope

This page covers credentials stored in:

It does not cover:

• TLS certificates issued by cert-manager — those rotate automatically under the cert-manager controller.
• Kubernetes ServiceAccount tokens issued by the projected-token mechanism — those rotate automatically.
• OAuth-issued JWTs / refresh tokens used by client applications — those rotate per the user-session lifecycle.

Pre-checks

Before mutating anything:

1. Identify the credential’s full consumer list. A single password is often consumed by multiple subsystems (a Nexus bot is used by Jenkins; the RHACS Central admin is used by automation to generate init-bundles; the WSO2 super-admin is consumed by both APIM and IS). Make the list explicit.

   # Example: find all references to the rotating credential's convention name in the workspace
   grep -rln "$CRED_NAME" /home/ze/opp-full-plat/ \
     | grep -v secrets/

2. Probe the current credential. Confirm the credential file is in fact current — sometimes “the credential is stale” really means “the live VM has the right value and the local mirror is out of date”.

   Follow the matrix in runbooks/secrets-custody-drift-check.md. Per service:

   Service	Probe
   Redis	redis-cli -h <host> --user <acl-user> -a "$PASS" PING -> expect PONG
   Nexus	curl -sS -u <user>:$PASS https://nexus-mirror.apps.sub.comptech-lab.com/service/rest/v1/security/users -o /dev/null -w "%{http_code}\n" -> expect 200
   WSO2	curl -sk -u admin:$PASS https://<wso2-host>:9443/services/UserAdmin?wsdl -o /dev/null -w "%{http_code}\n" -> expect 200
   Vault	VAULT_TOKEN=$PASS curl -sS $VAULT_ADDR/v1/auth/token/lookup-self -o /dev/null -w "%{http_code}\n" -> expect 200
   MinIO	mc alias set probe <endpoint> <ak> <sk> && mc ls probe/<bucket> > /dev/null -> expect zero exit

3. Open a GitHub issue describing the rotation: which credential, which consumers, the rotation window, and the validation plan. Routine rotations carry a secret-rot/ branch prefix.

4. Confirm the workspace boundary. Files under opp-full-plat/secrets/ are writable; do not put a new credential file anywhere else in /home/ze/.

The change

Five steps, in order. Skipping any one leaves the credential in a partially-rotated state.

Step 1 — Generate the new secret

The lab convention since the WSO2 incident: 24 chars, alphanumeric, no &, @, %, ?, :, #. These characters trigger URL-encoding traps across multiple subsystems (WSO2 JMS, basic-auth Authorization headers, S3 presigned URLs).

openssl rand -base64 30 | tr -d '/+=' | head -c 24
# k9JfWmZ7v3rN2bDqXcL5tHe8

tr -d '/+=' strips the base64 punctuation; head -c 24 truncates to the convention length. For higher-entropy use cases, raise the length, but keep the alphanumeric-only character set.

For RHACS init-bundles and similar structured tokens, follow the per-service generation API rather than rolling your own — see RHACS rotation.

Step 2 — Update the live service

The live service is the source of truth — change it there first. The exact mechanism depends on the service:

Capture the response — Vault tokens include an accessor, GitLab/GitHub PATs include an ID — for audit.

Step 3 — Update Vault (the canonical source for application credentials)

For tenant app secrets:

VAULT_ADDR=https://vault.sub.comptech-lab.com:8200
VAULT_TOKEN=<your-platform-admin-token>

vault kv put secret/apps/<division>/<app>/<env>/<key> \
  value="$NEW_VALUE"

For platform-scoped secrets (RHACS bundle, Quay config bundle, NooBaa keys consumed by ESO):

vault kv put secret/ocp/<scope>/<key> \
  value="$NEW_VALUE"

The path convention is documented in connection-details/vault-app-secrets.md. Always use the kv put flavour (not kv patch) when rotating, so the audit trail shows a clean overwrite rather than a partial update.

Step 4 — Update the local mirror

For credentials that also live in opp-full-plat/secrets/:

TS=$(date -u +%Y%m%dT%H%M%SZ)
mv `opp-full-plat/secrets/` (local-only, contents not enumerated here)<file> \
   `opp-full-plat/secrets/` (local-only, contents not enumerated here)<file>.pre-rotate-$TS
echo -n "$NEW_VALUE" > `opp-full-plat/secrets/` (local-only, contents not enumerated here)<file>
chmod 600 `opp-full-plat/secrets/` (local-only, contents not enumerated here)<file>

The .pre-rotate-<UTC> suffix preserves the prior value briefly — useful if validation fails and you need to roll back. Delete the backup file after the validation completes.

opp-full-plat/secrets/ is excluded from the workspace’s git status — files there are filesystem-only and never committed. The .gitignore at the workspace root enforces this; do not bypass it.

Step 5 — Trigger downstream consumers to pick up the new value

ExternalSecrets re-sync at their refreshInterval (default 5m or whatever the ES specifies). To force an immediate refresh on a tenant ES:

oc -n <tenant-ns> annotate externalsecret <es-name> \
  force-sync="$(date -u +%s)" --overwrite

For Jenkins / GitLab / GitHub PATs, restart any consumer service that caches the credential (or wait for its TTL). For RHACS init-bundles, the SecuredCluster on each managed cluster re-reads the Secrets that ESO recreates from Vault — see RHACS init-bundle rotation.

Service-specific flows

RHACS init-bundle

Generated via Central API rather than roxctl. The full flow:

KCFG="$HUB_KUBECONFIG"
ADMIN_PW=$(KUBECONFIG=$KCFG oc -n stackrox get secret central-htpasswd \
  -o jsonpath='{.data.password}' | base64 -d)
ROUTE=central-stackrox.apps.hub-dc-v6.sub.comptech-lab.com

curl -sk -u "admin:${ADMIN_PW}" -X POST \
  -H "Content-Type: application/json" \
  -d '{"name":"platform-cluster"}' \
  "https://${ROUTE}/v1/cluster-init/init-bundles" \
  > /tmp/bundle.json

The response includes kubectlBundle — base64-encoded YAML containing three Secrets (collector-tls, sensor-tls, admission-control-tls), each with stringData fields for ca.pem and the per-component cert + key.

Pipeline to flatten and push to Vault:

jq -r .kubectlBundle /tmp/bundle.json | base64 -d \
  | yq -r 'select(.kind == "Secret") | (.metadata.name + " " + (.stringData | to_entries | map(.key + "=" + .value) | join(" ")))' \
  | while read name pairs; do
      eval "set -- $pairs"
      for kv in "$@"; do
        K="${kv%%=*}"
        V="${kv#*=}"
        vault kv patch secret/ocp/platform/rhacs-init-bundle "$name.$K=$V"
      done
    done

The result is a Vault path secret/ocp/platform/rhacs-init-bundle with nine properties (3 Secrets x 3 keys each). The per-cluster ExternalSecrets in stackrox ns pull each property and recreate the original Secret. The bundle name (platform-cluster) appears as clusterName in Central UI for any SecuredCluster joined with this bundle; multiple SecuredClusters can use the same bundle.

The init-bundle expires (typical lifetime ~1 year — check expiresAt in the bundle metadata). Plan to rotate before expiration; the same flow regenerates and re-pushes to Vault, ExternalSecrets re-sync.

GitLab PAT (operator convention)

# 1. Open GitLab -> User Settings -> Access Tokens.
# 2. Note the existing PAT's ID; click "Revoke" after the new one is in hand.
# 3. Click "Add new token", scopes: api, write_repository, read_registry.
# 4. Copy the value; it is shown once.

TS=$(date -u +%Y%m%dT%H%M%SZ)
mv "$LOCAL_GITLAB_PAT_FILE"  # operator PAT, local-only \
   "$LOCAL_GITLAB_PAT_FILE.pre-rotate-$TS"
echo -n "<new-pat>" > "$LOCAL_GITLAB_PAT_FILE"  # operator PAT, local-only
chmod 600 "$LOCAL_GITLAB_PAT_FILE"  # operator PAT, local-only

# 5. Probe:
PAT=$(tr -d '
' < "$LOCAL_GITLAB_PAT_FILE")
curl -sSf -H "PRIVATE-TOKEN: $PAT" \
  http://<gitlab-vm>/api/v4/user | jq '.username'

# 6. Revoke the old PAT in the GitLab UI.
# 7. Delete the .pre-rotate-* backup file.

Nexus jenkinsbot password

# Generate a 24-char alphanumeric password (see convention above).
NEW_PASS=$(openssl rand -base64 30 | tr -d '/+=' | head -c 24)

# Update on Nexus via REST:
ADMIN_PASS=$(tr -d '\r\n' < "$NEXUS_ADMIN_PASSWORD")
curl -sSf -u "admin:$ADMIN_PASS" \
  -X PUT -H 'Content-Type: application/json' \
  -d "{\"password\":\"$NEW_PASS\"}" \
  https://nexus-mirror.apps.sub.comptech-lab.com/service/rest/v1/security/users/jenkinsbot/change-password

# Update local mirror:
TS=$(date -u +%Y%m%dT%H%M%SZ)
mv "$NEXUS_JENKINSBOT_PASSWORD" \
   "$NEXUS_JENKINSBOT_PASSWORD".pre-rotate-$TS
echo -n "$NEW_PASS" > "$NEXUS_JENKINSBOT_PASSWORD"
chmod 600 "$NEXUS_JENKINSBOT_PASSWORD"

# Probe:
curl -sS -u "jenkinsbot:$NEW_PASS" \
  https://nexus-mirror.apps.sub.comptech-lab.com/service/rest/v1/security/users \
  -o /dev/null -w "%{http_code}\n"
# Expect 200 (or 403 if the user lacks the API role but the credential is valid)

# Update Jenkins credentials store (UI or Jenkins CLI / curl):
# Manage Jenkins -> Credentials -> System -> Global -> jenkinsbot -> Update

Validation

A rotation is complete when all of the following are true:

1. The probe against the live service returns success (HTTP 200, PONG, etc.) with the new credential.
2. Every downstream consumer has been pinged with the new credential and returns success.
3. The local mirror file has been moved to <file>.pre-rotate-<UTC> and the new value is in <file>.
4. Vault (if applicable) has been overwritten with the new value.
5. The old credential has been revoked / disabled on the live service.
6. The pre-rotate backup file has been deleted (after step 5 succeeds).
7. The rotation has been recorded in the active session report under opp-full-plat/reports/sessions/ with file path, timestamp, actor, and probe evidence.

Skipping step 7 is what produces the next session’s “secrets drift” surprise.

Prevention

Three guardrails reduce the cost of rotation:

1. Use the safe character set every time. 24-char alphanumeric, no URL-special characters. The convention is documented at the top of this page and in runbooks/secrets-custody-drift-check.md. Tools that need higher-entropy keys (RHACS init-bundle TLS, Vault encryption keys) follow their service-specific generation API.

2. Probe before you rotate, not just after. Drift is bidirectional: the local mirror can drift from the VM, or vice versa. The drift-check runbook’s probe loop catches both directions. Add the drift check to the weekly cadence in day-1 handoff.

3. Use Vault as the canonical source for application credentials. Per-division Vault paths (secret/apps/<division>/<app>/<env>/*) and per-tenant SecretStores (namespace-scoped, never ClusterSecretStore) mean a rotation in Vault propagates to every consumer automatically. The local mirror remains the platform-credential backstop for credentials not yet in Vault (kubeadmin passwords, platform PATs, RHACS Central admin) — not a parallel source of truth.

Forbidden actions

• Updating the local mirror file with a password you did not also set on the live service. The file is a mirror, not the source of truth.
• Committing any secret file to Git. Filesystem-only custody is the convention; the .gitignore is the safety net.
• Using the admin credential to “punch through” a failing per-service probe — that defeats the probe.
• Proposing architecture / admin work that depends on a specific credential without first verifying it.
• Pasting the new credential value into chat, GitHub issues, GitLab MRs, the active session report, or anywhere else outside Vault and the local mirror.

References

• Runbook: opp-full-plat/runbooks/secrets-custody-drift-check.md
• opp-full-plat/connection-details/vault-app-secrets.md (per-division Vault path tree, policies, roles)
• opp-full-plat/connection-details/nexus.md
• opp-full-plat/connection-details/minio.md
• ADR: 0019-nexus-only-image-supply-chain, 0025-gitops-only-operations-break-glass