Installation Manual - 116 hub-dr-v7 deployment program plan
The milestone, workstream, phases, validation matrix, and reproducibility plan for the hub-dr-v7 warm standby deployment.
This chapter records the deployment program for hub-dr-v7, before live
deployment work begins.
The goal is a fully validated, documented, reproducible warm standby
management cluster for hub-dc-v7.
Governance
| Field | Value |
|---|---|
| Workstream | OP-GF-HUBDRV7 |
| Milestone | hub-dr-v7 Warm Standby Deployment |
| Parent issue | #448 / OP-GF-HUBDRV7-0 |
| Completed design issue | #446 / OP-GF-HUBDRV7-1 |
| Active next issue | #451 / OP-GF-HUBDRV7-5 |
| Governing ADRs | ADR 0016, ADR 0017, ADR 0024, ADR 0025, ADR 0029 |
| Local plan | plans/hub-dr-v7-warm-standby-deployment-program.md |
Certification Boundary
In this workstream, certified means internally certified for the CompTech lab:
- every phase has evidence;
- every live change has an issue and rollback path;
- every important state is reproducible from recorded inputs;
- OpenShift health, GitOps health, operator health, backup readiness, compliance scans, security checks, and failover readiness are validated;
- documentation exists locally and in this public installation manual.
This is not an external Red Hat certification, PCI DSS ROC/AOC, QSA, ISO, or bank-auditor attestation. External attestation requires a separate auditor-led track.
Non-negotiable Controls
hub-dr-v7stays warm standby until a failover issue explicitly promotes it.- Do not clone
hub-dc-v7identity: no copied etcd, cluster ID, kubeconfigs, OAuth secrets, service-account signers, RHACS runtime secrets, or ACM runtime state. - No live mutation without a phase issue, milestone, ADR coverage, validation plan, rollback path, and closeout.
- Do not print or commit secrets, kubeconfigs, pull secrets, PATs, Vault tokens, private keys, passwords, or full Secret manifests.
- Durable OpenShift state must be GitOps-first.
- Every material cluster gate must update this installation manual before closeout.
Target State
| Area | Target |
|---|---|
| Role | Warm standby management hub |
| Topology | Compact 3-node VM-only cluster |
| Release | Match hub-dc-v7, currently OpenShift 4.20.18 |
| Node role | control-plane,master,worker |
| VM spec | 32 vCPU, 64 GiB RAM per master |
| VM disks | 300 GiB boot + 3 x 500 GiB data per master |
| Candidate API/API-int VIP | 30.30.220.10 |
| Candidate ingress VIP | 30.30.220.11 |
| Candidate masters | 30.30.220.13-15 |
| Candidate MACs | 52:54:00:23:00:00-02 |
| Cluster network | 10.128.0.0/14, host prefix 23 |
| Service network | 172.30.0.0/16 |
Phase Line
The issue line remains granular for auditability. Execution and public
documentation from #451 onward should be batched so each article reads as a
complete runbook.
| Phase | Issue | Status | Gate |
|---|---|---|---|
OP-GF-HUBDRV7-0 | #448 | Open | Program ownership and certification evidence plan |
OP-GF-HUBDRV7-1 | #446 | Closed | Design and read-only preflight |
OP-GF-HUBDRV7-2 | #447 | Closed | Render install inputs and edge plan |
OP-GF-HUBDRV7-3 | #449 | Closed | Reserve IPAM, DNS, and HAProxy edge plan |
OP-GF-HUBDRV7-4 | #450 | Closed | Prepare VM env and libvirt dry-run |
OP-GF-HUBDRV7-5 | #451 | Open | Render agent ISO and create VMs |
OP-GF-HUBDRV7-6 | #452 | Open | Validate base OpenShift install |
OP-GF-HUBDRV7-7 | #453 | Open | Bootstrap GitOps, catalogs, mirrors, and storage foundation |
OP-GF-HUBDRV7-8 | #454 | Open | Install management operators in standby mode |
OP-GF-HUBDRV7-9 | #455 | Open | Prove backup, restore, and secret custody |
OP-GF-HUBDRV7-10 | #456 | Open | Close compliance and hardening evidence |
OP-GF-HUBDRV7-11 | #457 | Open | Operational readiness and SLO validation |
OP-GF-HUBDRV7-12 | #458 | Open | Write and dry-run failover runbook |
OP-GF-HUBDRV7-13 | #459 | Open | Controlled takeover drill |
OP-GF-HUBDRV7-14 | #460 | Open | Reproducibility certification evidence and closeout |
OP-GF-HUBDRV7-15 | #461 | Open | Full installation manual and handoff |
Batch Execution Model
The remaining work is grouped into four execution batches. Each batch may close multiple child issues, but the public installation manual should get one comprehensive article per batch.
| Batch | Issues | What It Proves |
|---|---|---|
| Prep batch, completed | #446, #447, #449, #450 | Design, install inputs, edge DNS, access, and VM dry-run readiness |
| Batch 1: Build and base cluster | #451, #452 | The VMs boot, the agent install completes, and base OpenShift health is clean |
| Batch 2: Standby platform foundation | #453, #454 | GitOps, catalogs, mirrors, storage, cert-manager, ESO/Vault auth, ACM/MCE, and RHACS are healthy without taking over spoke-dc-v7 |
| Batch 3: Protection and readiness | #455, #456, #457 | Backup/restore, secret custody, compliance, hardening, monitoring, alerting, logging, and SLO checks are recorded |
| Batch 4: DR certification and handoff | #458, #459, #460, #461 | Failover runbook, controlled takeover drill, reproducibility evidence, and full handoff are complete |
Future articles should include objective, non-goals, entry criteria, execution sequence, rollback, validation matrix, secrets handling, GitOps/live deltas, residual risks, and the next-batch decision.
Gate Model
Every phase must close with:
- issue comment with what changed;
- validation evidence;
- files touched;
- live systems touched;
- rollback or promotion decision;
- residual risks;
- next gate;
- local session record;
- updated local memory files;
- blog chapter or blog update when the gate is material.
Validation Matrix
| Domain | Required evidence |
|---|---|
| Governance | Issue, milestone, phase, ADR coverage |
| Network | IPAM, DNS, HAProxy validation and rollback |
| VM | Libvirt domain, CPU, RAM, disk, NIC, boot state |
| Install inputs | install-config, agent-config, mirror resources, FIPS, host count |
| Base cluster | ClusterVersion, nodes, MCPs, ClusterOperators, CSRs, pods |
| GitOps | Root app, revision, sync, health, render validation |
| Operators | Subscription, InstallPlan, CSV, operand readiness |
| Secrets | Vault/ESO path and readiness, no value disclosure |
| Backup | DPA, BSL, schedule, backup, restore or dry-run evidence |
| Security | RHACS health, Gatekeeper health, NetworkPolicies, image policy |
| Compliance | CIS, PCI DSS 4, File Integrity, exception register |
| Operations | API, console, ingress, monitoring, logging, alerting |
| DR | Failover runbook, dry-run, takeover drill, rollback |
| Documentation | Local report, session log, runbook, blog chapter |
| Reproducibility | Source paths, commits, commands, artifacts, rebuild notes |
Final Done Criteria
hub-dr-v7 is complete only when all are true:
- cluster is installed and healthy;
- platform GitOps is synced and healthy;
- management services are installed in standby-safe mode;
- backup, restore readiness, and secret custody are proven;
- compliance and hardening evidence is closed or explicitly excepted;
- operational readiness checks pass;
- failover runbook dry-run is complete;
- controlled takeover drill is completed or explicitly deferred;
- public installation manual and local runbooks are complete;
- reproducibility evidence pack is complete.
Next Gate
The next execution unit is Batch 1:
#451 / OP-GF-HUBDRV7-5: render hub-dr-v7 agent ISO and create VMs
#452 / OP-GF-HUBDRV7-6: validate base hub-dr-v7 OpenShift install
The closeout should be one comprehensive build and base-install article, not two thin gate notes.