Routine tasks — overview
Nine recurring operator workflows that own the fleet's running state: secret rotation, operator bumps, fleet onboarding, policy rollout, evidence backfill, kubeadmin and RHACS Central rotation, division onboarding, Loki OBC bridge.
This subsection is the operator’s playbook for the recurring work the fleet pays for during steady state. Each task is small enough to complete in one session, common enough to standardise, and important enough to get wrong if rushed. Read every page once; come back when the task is on your plate.
Every routine task lands as a platform-gitops MR. There are no exceptions — secret rotations land via ESO config changes, evidence backfills land via lifecycle rule updates, even RHACS init-bundle rotations land via Vault writes + ExternalSecret refreshes captured in GitOps. The MR loop is the standardisation; the per-task page tells you what to put in the MR.
The five-step shape
Every page in this subsection follows the same shape:
- When it runs. Cadence (calendar-driven, alert-driven, install-driven) and the trigger that surfaces the task on TODO.
- What is in scope. The specific systems/credentials/files touched. Boundaries against adjacent tasks.
- Pre-checks. The state you confirm before you start mutating. Always read-only.
- The change. The actual MR / commands / config that implement the rotation/bump/onboarding.
- Validation. The end-of-task evidence that the fleet returned to steady state.
If you find yourself reading a task page and it does not have those five sections, the page is incomplete — file an issue under #229 against this section.
The nine routine tasks
| # | Page | Cadence | Triggers |
|---|---|---|---|
| 02 | Rotate secrets and tokens | Quarterly or on personnel change | Calendar; offboarding event; drift-check failure |
| 03 | Bump operator version | When upstream releases a CSV with a security fix or required feature | CSV release; CVE; ADR-mandated feature |
| 04 | Add a cluster to the fleet | On capacity event or DR build-out | New ManagedCluster onboarding |
| 05 | Roll out a policy | On compliance gate or new finding | PCI-DSS scan finding; ACM PolicySet update |
| 06 | Backfill evidence to MinIO | When an evidence-pack window closes | CI run; compliance close-out; lifecycle audit |
| 07 | Rotate the kubeadmin password | Quarterly or on personnel change | OCP 4.20 23-char minimum; htpasswd-Secret recreate; Vault custody |
| 08 | Rotate the RHACS Central admin | Quarterly or after init-bundle regen | Vault secret/ocp/platform/rhacs-admin -> ESO -> rollout restart deploy/central |
| 09 | Add a division to federated GitLab | On new client engagement or compliance scope-change | ct-* role groups; CODEOWNERS; runner-class tags; Vault path tree |
| 10 | Publish the Loki OBC bridge | One-shot under #233 | LokiStack stuck Warning Degraded; backport of the Tempo bridge !43 |
The cadence column matters: tasks 02 / 06 / 07 / 08 are calendar-driven; tasks 03 / 04 / 09 are event-driven; tasks 05 / 10 are gate-driven (compliance and incident-follow-up respectively). The on-call rotation escalation matrix tells you who owns each.
What is not in this subsection
- Incidents (live failures) live in the incidents subsection. Routine tasks run during steady state; incidents run during not-steady state.
- Cluster upgrades (OCP minor/patch). Upgrade procedures are governed by ADR 0018’s pull model and the OCP upgrade documentation. They are not yet a published runbook on this site; the next operator who runs one should write the page.
- OADP backup drills. Backup/restore drills follow the OADP operator’s documentation plus the lab’s evidence-capture convention. A drill runbook is planned under #229 follow-ups.
- Trivy scanner refresh and CI evidence rotation. These touch the developer-side pipeline (GitLab -> Jenkins -> Trivy -> Nexus -> Docker runtime VM) which is paused for app-delivery scope per the user’s 2026-05-09 decision (
project_app_dev_direction). When OpenShift app delivery is reopened, these will join routine tasks.
A note on naming
The session-handoff convention is one routine task = one MR = one issue = one session report. Branch name embeds the issue key. The MR description references the routine-task page on this site (this is the public-facing “why”). The session report captures the validation evidence.
Naming patterns observed in the active repo:
| Task class | Branch prefix | Example |
|---|---|---|
| Secret rotation | secret-rot/ | secret-rot/gitlab-pat-rotate-202605 |
| Operator bump | op-bump/ | op-bump/oadp-1.5.5-to-1.5.6 |
| Cluster onboarding | cluster-onb/ | cluster-onb/spoke-dr-v6-import |
| Policy rollout | policy/ | policy/pci-dss-4-allowedregistries |
| Evidence backfill | evidence/ | evidence/pci-baseline-2026-05-11 |
These prefixes are not enforced by any hook — they are the lab’s conventional shape.
References
opp-full-plat/connection-details/platform-admin-handoff.mdopp-full-plat/runbooks/secrets-custody-drift-check.mdopp-full-plat/plans/disconnected-rebuild/environments/dc-lab/operator-version-lock.md- Issues: #229 (this section), #137 (hub mirror capture — example of evidence-backfill class), #233 (Loki OBC bridge backport), #255 / MR
!73(RHACS Central admin via Vault + ESO)