Rotate the kubeadmin password

Why OCP 4.20 rejects a kubeadmin password under 23 characters even when the bcrypt hash matches, why `oc adm reset-password` doesn't exist, and the htpasswd-Secret recreation procedure plus Vault custody update.

This page covers rotating the kubeadmin password on an OCP 4.20 cluster. The procedure is short, but it has one load-bearing gotcha — OCP 4.20 enforces a 23-character minimum on the kubeadmin password, and the kube:admin identity provider rejects logins under 23 chars even when the bcrypt hash in the Secret matches the password. This is specific to kubeadmin; htpasswd IdP users are unaffected. The other gotcha — oc adm reset-password kubeadmin does not exist on OCP 4.20 — means rotation is done by recreating the kubeadmin Secret in kube-system with a freshly-generated bcrypt hash. Treat this as a kubeadmin-only page; for fleet-wide secret rotation, Rotate secrets and tokens is the parent procedure.

When to run this

Quarterly cadence under the lab’s 90-day rotation policy for long-lived platform credentials.
Personnel change — someone with kubeadmin access leaves the team.
Suspected leak — the password landed in chat, a session report, or a commit message.
Lifecycle event — fresh cluster install (the installer generates kubeadmin; rotate before handing off to a different operator).
Migration step — when an htpasswd IdP is being added or replaced, validate kubeadmin still works before moving on.

Prerequisites

A working break-glass IdP. If the rotation goes wrong and you lock yourself out, the htpasswd IdP (or another OAuth IdP) is the only way back in. Confirm before rotating:
```
K=/home/ze/.kube/configs/<cluster>.kubeconfig
oc --kubeconfig "$K" get oauth cluster -o yaml \
  | grep -A5 identityProviders
```
At least one non-kube:admin IdP must be configured and an account in it must have cluster-admin. If not, do not rotate kubeadmin — fix the IdP first.
htpasswd binary on the operator workstation for hashing the new password:
```
which htpasswd || sudo dnf install -y httpd-tools
```

Vault write access for updating custody after the rotation:

VAULT_ADDR=https://vault.sub.comptech-lab.com:8200
vault token lookup-self

A GitHub issue describing the rotation; branch prefix secret-rot/kubeadmin-<cluster>-<UTC-date>.

Procedure

The full flow is four steps: generate, recreate Secret, validate login, update Vault custody.

Step 1 — Generate a new password (at least 23 characters)

NEW_PASS=$(openssl rand -base64 30 | tr -d '/+=' | head -c 28)
echo "$NEW_PASS"

The convention is 24 alphanumeric characters minimum, no &, @, %, ?, :, # (these trigger URL-encoding traps in adjacent subsystems). The OCP 4.20 floor is 23 characters — anything under 23 will be silently rejected at login time even though the Secret accepts it. Use 28 to leave headroom.

If the password is generated under 23 chars, the symptom is confusing: oc login -u kubeadmin -p <pass> returns Login failed (401 Unauthorized) even though the bcrypt comparison matches. The provider has a length check before the bcrypt check. There is no error message that points at the length.

Step 2 — Hash the password and recreate the kubeadmin Secret

oc adm reset-password kubeadmin does not exist on OCP 4.20. The procedure is to recreate the Secret directly:

NEW_HASH=$(htpasswd -nbBC 10 "" "$NEW_PASS" | tr -d ':\n')

oc --kubeconfig "$K" -n kube-system delete secret kubeadmin

oc --kubeconfig "$K" -n kube-system create secret generic kubeadmin \
  --from-literal=kubeadmin="$NEW_HASH"

The Secret’s single key kubeadmin holds the bcrypt hash. htpasswd -nbBC 10 emits :<hash>; the tr -d ':\n' strips the leading colon and trailing newline so the value stored is the bare hash.

The authentication-operator picks up the new Secret within seconds — there is no need to restart any pod.

Step 3 — Validate the new password

Wait a few seconds for the authentication-operator to reconcile, then test:

oc login --kubeconfig=/tmp/kubeadmin-test.kubeconfig \
  -u kubeadmin -p "$NEW_PASS" \
  https://api.<cluster>.sub.comptech-lab.com:6443

oc --kubeconfig=/tmp/kubeadmin-test.kubeconfig get clusterversion

Expected: successful login, clusterversion shows the cluster. If login fails with 401:

Confirm the password is >= 23 chars (re-check with echo -n "$NEW_PASS" | wc -c).
Confirm the Secret was created in the right namespace (kube-system, not the cluster’s default).
Wait another 30 seconds for reconcile and retry.
If still failing, fall back to the htpasswd break-glass IdP (see Break-glass fallback).

Delete the test kubeconfig after validation:

rm /tmp/kubeadmin-test.kubeconfig

Step 4 — Update Vault and the local mirror

TS=$(date -u +%Y%m%dT%H%M%SZ)

vault kv put secret/ocp/<cluster>/kubeadmin \
  password="$NEW_PASS"

KUBEADMIN_FILE=/home/ze/opp-full-plat/secrets/<cluster>-kubeadmin-password
mv "$KUBEADMIN_FILE" "$KUBEADMIN_FILE.pre-rotate-$TS"
echo -n "$NEW_PASS" > "$KUBEADMIN_FILE"
chmod 600 "$KUBEADMIN_FILE"

unset NEW_PASS NEW_HASH

Vault is the canonical source. The local mirror under opp-full-plat/secrets/ is the backstop for break-glass — if Vault is itself unreachable, the operator workstation still has the password. Delete the .pre-rotate-<TS> backup after the rotation has been validated for at least 24 hours.

Verification

The rotation is complete when ALL of these are true:

oc login -u kubeadmin -p <new> succeeds against the cluster’s kube-apiserver and oc get clusterversion returns the cluster.
oc login -u kubeadmin -p <old> fails with 401 (proves the old password is dead, not just that the new one works).
vault kv get secret/ocp/<cluster>/kubeadmin returns the new password.
The local mirror file contains the new password and the .pre-rotate-<TS> backup is in place.
The session report or incident issue captures: cluster name, UTC timestamp, actor, Vault path, and the validation evidence (the successful login on the new password).

Skip step 5 and the next session’s “kubeadmin drift” surprise is on you.

Break-glass fallback

If the rotation goes wrong and oc login -u kubeadmin no longer works:

Use the htpasswd break-glass IdP. The IdP user break-glass is configured on every lab cluster with cluster-admin. Its password is in opp-full-plat/secrets/<cluster>-break-glass-password.

BG_PASS=$(tr -d '\r\n' < /home/ze/opp-full-plat/secrets/<cluster>-break-glass-password)
oc login -u break-glass -p "$BG_PASS" \
  https://api.<cluster>.sub.comptech-lab.com:6443

From the break-glass session, fix the kubeadmin Secret. Recreate it with a known-good hash from a password that meets the 23-char minimum.
Open a break-glass audit record under opp-full-plat/reports/break-glass/ per the Break-glass procedure. The rotation that needed the fallback is itself a break-glass event because GitOps cannot recover from a broken kubeadmin.

Re-verify the htpasswd IdP after recovery. Confirm break-glass still works for next time:

oc login -u break-glass -p "$BG_PASS" --kubeconfig=/tmp/bg.kubeconfig \
  https://api.<cluster>.sub.comptech-lab.com:6443
rm /tmp/bg.kubeconfig

Forbidden actions

Do NOT generate a password under 23 characters. OCP 4.20’s kube:admin provider rejects it at login even though the Secret accepts it. The error message does not point at the length.
Do NOT skip the validation step. A successful Secret recreate does not prove the new password works. Test the login.
Do NOT leave the .pre-rotate-<TS> backup file in place indefinitely. Delete it once the new password has been validated for at least 24 hours.
Do NOT commit any password value to Git. The local mirror is .gitignore-excluded; do not bypass.
Do NOT paste the new password into a chat, session report, GitHub issue, or any non-Vault, non-mirror destination.

References

Rotate secrets and tokens — the parent rotation procedure; this page is the kubeadmin-specific overlay.
Break-glass procedure — fallback when the rotation breaks login.
Secrets custody drift check — probe matrix to confirm the new password before declaring done.
opp-full-plat/connection-details/platform-admin-handoff.md — kubeadmin custody and the htpasswd break-glass IdP convention.
OCP 4.20 release notes — kubeadmin minimum-length enforcement.