Installation Manual - 125 hub-dr-v7 failover rehearsal certification and handoff

Batch 4 execution for hub-dr-v7: failover runbook dry-run, actual spoke-dc-v7 takeover, certification evidence, and operator handoff.

This chapter records Batch 4 of the hub-dr-v7 warm-standby management hub deployment. It combines issue #458, issue #459, issue #460, and issue #461 into one operational evidence set.

The first pass through this chapter was deliberately conservative: the failover runbook was written and server-side dry-run validation passed, but the actual takeover was stopped because hub-dc-v7 was healthy and still authoritative for spoke-dc-v7.

The operator later explicitly requested the actual failover. On 2026-05-22T05:54:43Z, spoke-dc-v7 management authority was moved to hub-dr-v7, and the stale hub-dc-v7 spoke authority was removed.

Governance

Objective

Close the safe portion of the final hub-dr-v7 batch:

• write a usable failover runbook from hub-dc-v7 to hub-dr-v7;
• dry-run the takeover object model without transferring authority;
• validate that the current active and standby hub boundaries are clean;
• decide whether controlled takeover is allowed under the issue entry criteria;
• record reproducibility and operator handoff evidence.

Non-Goals

• Do not promote hub-dr-v7 while hub-dc-v7 is healthy.
• Do not import or actively manage spoke-dc-v7 from hub-dr-v7 unless a deliberate failure or isolation condition is declared.
• Do not create a second ApplicationSet or GitOpsCluster for spoke-dc-v7 while the active hub still owns it.
• Do not publish kubeconfigs, kubeadmin passwords, pull secrets, Vault tokens, object-storage credentials, RHACS credentials, HTPasswd values, private keys, PATs, or full Secret manifests.
• Do not present this internal certification as external Red Hat, PCI DSS ROC/AOC, QSA, ISO, or bank-auditor attestation.

Entry State

Batch 3 left hub-dr-v7 operationally ready:

• OpenShift 4.20.18;
• three compact masters Ready;
• GitOps root app healthy;
• management operators installed;
• Vault-backed identity in place;
• kubeadmin retired after replacement login validation;
• OADP backup and restore smoke test complete;
• RHACS Central and SecuredCluster available;
• Compliance Operator CIS and PCI DSS 4 scans clean;
• File Integrity node status clean;
• standby boundary intact.

The final standby boundary before Batch 4 was:

hub-dr-v7:
  ManagedCluster/local-cluster only
  no ApplicationSet
  no GitOpsCluster
  no ManagedCluster/spoke-dc-v7

Stop Conditions

The batch had to stop before takeover if any of these occurred:

• hub-dc-v7 remained healthy and authoritative;
• hub-dc-v7 failure or isolation was not deliberately declared;
• spoke-dc-v7 was still reconciled by the active hub;
• server-side dry-run failed for the candidate takeover objects;
• backup posture could not be checked at a useful summary level;
• any command path would print secrets or full Secret manifests.

The first two stop conditions were true, so actual takeover was correctly blocked.

Live Systems Touched

Read-only and dry-run checks touched:

• hub-dc-v7;
• spoke-dc-v7;
• hub-dr-v7;
• gf-ocp-bootstrap-01;
• dl385-2.

No durable OpenShift object was created, changed, or deleted by the takeover dry-run.

Execution Sequence

The batch started with the local workspace state, issue bodies, and the latest session records. The next work was already governed by milestone #34, parent issue #448, child issues #458 through #461, and ADRs 0016, 0017, 0024, 0025, and 0029.

The live checks were run through the governed bootstrap path:

ssh ze@dl385-2 'ssh gf-ocp-bootstrap-01'

The kubeconfigs remained on the bootstrap host under:

/home/ze/ocp-greenfield-deployment/artifacts/openshift/<cluster>/auth/kubeconfig

The validation sequence was:

1. Check ClusterVersion, nodes, ClusterOperators, and non-running pods for hub-dc-v7, spoke-dc-v7, and hub-dr-v7.
2. Check active management authority on hub-dc-v7.
3. Check standby authority absence on hub-dr-v7.
4. Confirm required takeover CRDs and API resources exist on hub-dr-v7.
5. Run server-side dry-run for takeover candidate resources.
6. Decide go or no-go for actual takeover based on #459 entry criteria.
7. Update local runbooks, evidence, state files, and this installation-manual chapter.

Live Validation

The final live validation at 2026-05-21T22:06:05Z showed all three clusters healthy.

Authority Boundary

hub-dc-v7 remained active management authority:

ManagedCluster/local-cluster
ManagedCluster/spoke-dc-v7
ApplicationSet/openshift-gitops/spoke-cluster-config-pull
GitOpsCluster/openshift-gitops/gitops-managed

hub-dr-v7 remained standby-only:

ManagedCluster/local-cluster
no ApplicationSet
no GitOpsCluster
no ManagedCluster/spoke-dc-v7

This is the expected normal operating state. It prevents two hubs from reconciling the same workload cluster.

Dry-Run Candidate

The dry-run candidate checked the minimum object model needed for a future controlled promotion:

• namespace spoke-dc-v7;
• ManagedCluster/spoke-dc-v7;
• Placement/openshift-gitops/gitops-managed;
• GitOpsCluster/openshift-gitops/gitops-managed.

hub-dr-v7 exposes the required API resources:

ManagedCluster
ManagedClusterSet
ManagedClusterSetBinding
Placement
GitOpsCluster
ApplicationSet

Server-side dry-run passed for the namespace, ManagedCluster, Placement, and GitOpsCluster candidates. The dry-run did not create the resources.

Controlled Takeover Decision

Issue #459 requires a deliberately declared hub-dc-v7 failure or isolation condition before takeover.

The observed condition was:

• hub-dc-v7 healthy;
• spoke-dc-v7 healthy;
• hub-dc-v7 still actively owns spoke-dc-v7;
• no failure or isolation declared.

Decision: no-go for actual takeover.

This is not a failed drill. It is the correct safety outcome. Promoting hub-dr-v7 while the active hub is healthy would create avoidable split-brain risk and violate the warm-standby design in ADR 0029.

Actual Takeover Execution

After the dry-run/no-go result, the operator explicitly requested actual failover/takeover of spoke-dc-v7 by hub-dr-v7. That request was treated as the planned failover/isolation window for issue #459.

The execution sequence was:

1. Remove active hub GitOps authority for spoke-dc-v7.
2. Add equivalent management authority on hub-dr-v7.
3. Sync both hub root Argo CD applications.
4. Apply the hub-dr-v7 generated ACM import CRDs and import payload directly to spoke-dc-v7 without printing the payload.
5. Prune the stale active-hub ManagedCluster/spoke-dc-v7 and namespace from hub-dc-v7.
6. Validate final authority, cluster health, addons, and ManifestWorks.

Platform GitOps commits:

a822c3e Move spoke management authority to hub-dr-v7
0c7293c Prune spoke registration from active hub

Final authority state:

hub-dc-v7:
  ManagedCluster/local-cluster only
  no ApplicationSet/spoke-cluster-config-pull
  no GitOpsCluster/gitops-managed
  no ManagedCluster/spoke-dc-v7

hub-dr-v7:
  ManagedCluster/local-cluster
  ManagedCluster/spoke-dc-v7 Joined=True Available=True
  ApplicationSet/openshift-gitops/spoke-cluster-config-pull
  GitOpsCluster/openshift-gitops/gitops-managed

spoke-dc-v7 Klusterlet validation:

available=True
applied=True

All hub-dr-v7 ManifestWorks in namespace spoke-dc-v7 were Applied=True and Available=True, including the generated spoke-dc-v7-cluster-config ManifestWork.

The generated Argo Application spoke-dc-v7-cluster-config remains OutOfSync and Healthy because it is intentionally annotated with:

argocd.argoproj.io/skip-reconcile=true

For this OCM pull model, the ManifestWork status is the authoritative validation signal.

The retain-or-rollback decision was:

retain hub-dr-v7 authority

No split-brain condition remained at closeout.

Initial Post-Takeover Soak Watch

The first post-takeover watch was recorded under issue #462 OP-GF-HUBDRV7-16: post-takeover soak watch and health validation.

Watch window:

2026-05-22T06:12:00Z through 2026-05-22T06:18:27Z

Six read-only samples were captured about one minute apart. Every sample kept the failover-specific signals clean:

The generated Application/spoke-dc-v7-cluster-config on hub-dr-v7 remained:

OutOfSync Healthy skip-reconcile=true

That is expected for this OCM pull model. ManifestWork and Klusterlet status remain the authoritative validation signals.

The first watch loop counted completed job pods as non-running, so the pod phase output was classified separately:

Backup and policy surfaces were also checked:

• DataProtectionApplication/openshift-adp/hub-dr-v7 was Reconciled=True.
• BackupStorageLocation/openshift-adp/hub-dr-v7 was Available.
• Schedule/openshift-adp/platform-resource-daily was Enabled, with the latest backup about 4h2m old during the observation.
• DataProtectionApplication/openshift-adp/spoke-dc-v7 was Reconciled=True.
• BackupStorageLocation/openshift-adp/spoke-dc-v7 was Available.
• No hub-dr-v7 policies outside Compliant were listed.
• Placement/openshift-gitops/gitops-managed selected one cluster.

Result:

initial post-takeover soak clean

Failover Runbook

The failover runbook now records:

• promotion entry criteria;
• read-only prerequisite checks;
• backup freshness checks;
• isolation decision path;
• activation candidate sequence;
• service validation after promotion;
• rollback order;
• the 2026-05-21 dry-run result.

The runbook is local to the operations workspace:

plans/hub-dr-v7-failover-runbook.md

The central rule is:

If hub-dc-v7 is healthy and reachable, the correct drill result is no-go.

Reproducibility Matrix

Certification Boundary

hub-dr-v7 is internally certified as a protected warm-standby management hub for the CompTech lab and, after the 2026-05-22 takeover, as the active management authority for spoke-dc-v7.

The certification covers:

• healthy OpenShift 4.20.18 compact cluster;
• GitOps-managed desired state;
• disconnected catalog and mirror policy posture;
• LVMS storage foundation;
• cert-manager, External Secrets, Vault integration, ACM/MCE, RHACS, OADP, Gatekeeper, logging, Compliance Operator, and File Integrity Operator;
• Vault-backed break-glass identity and kubeadmin removal;
• OADP restore proof;
• RHACS backup evidence;
• selected CIS and PCI DSS 4 Compliance Operator scans clean;
• failover runbook and server-side dry-run.

The certification does not cover:

• formal external compliance attestation;
• active-active hub operation.

Operator Handoff

Normal operation after takeover:

1. Treat hub-dr-v7 as the active management hub for spoke-dc-v7.
2. Treat hub-dc-v7 as non-authoritative for spoke-dc-v7.
3. Monitor health, GitOps sync, OADP schedules, RHACS, Compliance Operator, File Integrity, and Vault/ESO readiness.
4. Do not recreate spoke-dc-v7 import resources on hub-dc-v7 unless a separate governed failback is approved.

Incident operation:

1. Declare hub-dc-v7 failed or isolated.
2. Confirm split-brain prevention.
3. Check backup freshness and spoke-dc-v7 health.
4. Follow plans/hub-dr-v7-failover-runbook.md.
5. Validate authority, GitOps, RHACS, OADP, logging, compliance, and workload reachability after promotion.
6. Decide to roll back or retain hub-dr-v7 authority from evidence.

Residual Risks

• Reverse failback to hub-dc-v7 remains unexecuted.
• hub-dr-v7 is now the active management authority for spoke-dc-v7; future operations must not assume hub-dc-v7 still owns that spoke.
• External audit attestation remains out of scope for this lab certification.

Closeout

Batch 4 first reached a dry-run safe stop, then completed the actual takeover:

• failover runbook written;
• dry-run validation passed;
• active and standby authority boundaries proven;
• spoke-dc-v7 reimported to hub-dr-v7;
• stale hub-dc-v7 spoke authority removed;
• split-brain risk avoided and then eliminated;
• handoff evidence recorded.

The fleet should now operate with hub-dr-v7 as active management authority for spoke-dc-v7 until a separate governed failback moves authority back.