Installation Manual - 123 hub-dr-v7 standby platform foundation and operators

Batch 2 execution for hub-dr-v7: GitOps bootstrap, catalogs, mirrors, storage, Vault-backed secret sync, ACM/MCE, RHACS, OADP, Gatekeeper, logging, Compliance, and File Integrity.

This chapter records Batch 2 of the hub-dr-v7 warm-standby management hub deployment. It combines issue #453 and issue #454 into one rebuildable manual page: bootstrap platform GitOps, wire disconnected catalogs and image mirrors, install the storage and secret foundation, install the management operators, validate the operands, and prove that the DR hub has not taken over spoke-dc-v7.

Governance

Objective

Turn the clean base hub-dr-v7 OpenShift install into an operator-ready standby management hub.

Target state:

root GitOps app: hub-dr-v7-bootstrap
catalog model: disconnected Red Hat and certified v4.20 catalogs
storage: LVMS with lvms-vg1 default StorageClass
secret source: R1 Vault through External Secrets Operator
management: ACM/MCE installed for standby hub use
security: RHACS Central/SecuredCluster, Gatekeeper, Compliance, File Integrity
backup: OADP DPA, BSL, and daily platform-resource schedule
logging: LokiStack and ClusterLogForwarder
standby boundary: no spoke-dc-v7 import or takeover resources

Non-Goals

• Do not import spoke-dc-v7.
• Do not enable a hub-dr-v7 ApplicationSet that can prune resources managed by hub-dc-v7.
• Do not promote hub-dr-v7 as the active management hub.
• Do not execute a failover drill.
• Do not claim CIS or PCI DSS 4 compliance closure in this batch.
• Do not publish kubeconfigs, pull secrets, Vault tokens, object-storage credentials, RHACS init-bundle values, PATs, passwords, private keys, or full Secret manifests.

Entry Criteria

Batch 1 had to be complete:

• hub-dr-v7 installed on OpenShift 4.20.18;
• all three compact masters Ready;
• all ClusterOperators green;
• master MCP updated;
• no pods outside Running/Succeeded;
• API, ingress, console, DNS, and VM boot state validated.

Before mutation, the batch rechecked the base cluster and confirmed the governance objects for #453, #454, milestone hub-dr-v7 Warm Standby Deployment, and ADR coverage.

GitOps Desired State

The platform desired state lives in:

git@github.com:zeshaq/openshift-platform-gitops.git
clusters/hub-dr-v7

Commits pushed during this batch:

0de88af Add hub-dr-v7 standby platform GitOps
d32e0ad Fix hub-dr-v7 Argo namespace bootstrap order
364f4bc Allow hub-dr-v7 CR sync before CRDs settle
472a517 Fix hub-dr-v7 RHACS secret store sync order
a6254d8 Stage hub-dr-v7 RHACS secured-cluster after Central
04a2edc Let MCE own hub-dr-v7 local cluster namespace
60bc2b8 Record hub-dr-v7 standby platform batch

The root app finished at:

Synced Healthy Succeeded
revision: 04a2edc90bea5b584dd08101c01e32d2ef397d13

The extra 60bc2b8 commit updates the platform GitOps changelog and does not change the rendered clusters/hub-dr-v7 payload.

Execution

High-level sequence:

1. Clone the active platform GitOps repository locally.
2. Create clusters/hub-dr-v7/ from the working hub-dc-v7 profile.
3. Remove active spoke registration and takeover resources.
4. Add scripts/bootstrap-hub-dr-v7-gitops.sh.
5. Validate oc kustomize clusters/hub-dr-v7.
6. Validate oc kustomize clusters/hub-dr-v7/gitops-control.
7. Push the desired state.
8. Pull the new desired state on gf-ocp-bootstrap-01.
9. Seed required non-published values in R1 Vault for ESO, RHACS, OADP, and logging.
10. Configure Vault Kubernetes auth for hub-dr-v7.
11. Run the bootstrap script to install OpenShift GitOps and seed the root Application.
12. Fix Argo ordering issues discovered during the first convergence.
13. Generate an independent RHACS init bundle from the new hub-dr-v7 Central and store the required values in Vault.
14. Remove the GitOps-created local-cluster namespace so MCE can own self-management.
15. Re-run the full validation pass after convergence.

Ordering Corrections

Four ordering corrections became part of the final desired state.

Namespace Bootstrap

Argo initially tried to apply namespaced resources before some namespaces existed. The correction added a bootstrap namespace bundle at sync wave -20.

Operator CRDs

Several operator-owned custom resources are applied in the same root app as their operators. Those resources now use:

argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

This allows Argo to continue while operator CRDs settle.

RHACS Secret Staging

RHACS now converges in this order:

Vault ClusterSecretStore
central-admin-password ExternalSecret
Central
init-bundle ExternalSecrets
SecuredCluster

The init bundle was generated from the new Central endpoint for this cluster. Only decoded key/value material was written to Vault. No secret value or full Secret manifest is published here.

MCE Local Cluster Ownership

The first namespace bootstrap included local-cluster. MCE waits for that namespace to be absent before it creates the self-managed ManagedCluster/local-cluster, so MCE remained Progressing.

The correction:

• removed local-cluster from the GitOps-created namespace bundle;
• deleted the pre-created live namespace;
• let MCE recreate it through its own self-management path;
• moved the local-cluster NetworkPolicy to sync wave 90.

After that, MCE became Available and MCH became Running.

Vault And Secret Sync

R1 Vault now has hub-dr-v7 Kubernetes auth coverage for:

kubernetes-hub-dr-v7
eso-secrets
rhacs-secrets
oadp-backup
logging-loki

Final secret-sync validation:

Operator Versions

All subscriptions reported AtLatestKnown, and every CSV reached Succeeded.

Operand Validation

Compliance Scan State

Compliance ran successfully, but the cluster is not compliant yet.

ocp4-cis: DONE / NON-COMPLIANT
ocp4-cis-node-master: DONE / COMPLIANT
ocp4-pci-dss-4-0-hub-dr-v7: DONE / NON-COMPLIANT
ocp4-pci-dss-node-4-0-master: DONE / COMPLIANT

This is expected for Batch 2. The next batch owns remediation, tailoring, evidence, and operational acceptance.

Standby Safety

Final standby checks:

hub-dr-v7 is now self-managed, which is normal for ACM/MCE, but it is not managing the active workload cluster.

Secrets Handling

No kubeconfig, kubeadmin password, pull secret, PAT, private key, token, object-storage credential, RHACS init-bundle value, or full Secret manifest is published in this chapter.

One transient RHACS init-bundle file existed only long enough to convert the generated bundle into Vault key/value entries. It was removed after the values were written.

Rollback Notes

The GitOps rollback unit for this batch is the clusters/hub-dr-v7/ desired state and the root Argo application.

Safe rollback principles:

• do not delete or alter hub-dc-v7;
• do not delete or alter spoke-dc-v7;
• do not remove Vault global auth or shared paths outside the hub-dr-v7 scope;
• remove or revert only hub-dr-v7-specific GitOps resources;
• preserve install artifacts under the governed hub-dr-v7 artifact path unless a later destroy gate explicitly removes the cluster.

Residual Risks

• CIS and PCI DSS 4 platform scans are non-compliant.
• OADP has an enabled daily schedule and an available BSL, but restore has not been drilled yet.
• Logging uses an existing greenfield MinIO bucket/credential as a temporary unblocker. A dedicated Loki bucket/user should be created in the custody/backup batch.
• Kubeadmin retirement, identity-provider integration, and pull-secret custody proof remain open.
• No failover or controlled spoke-dc-v7 takeover has been rehearsed.

Exit State

hub-dr-v7 is now a GitOps-managed, operator-ready warm-standby management hub. It is not yet production-ready for failover because backup restore, custody closure, compliance hardening, operational readiness, and takeover drills are still pending.

Next Batch

Proceed to:

#455 OP-GF-HUBDRV7-9: prove hub-dr-v7 backup restore and secret custody
#456 OP-GF-HUBDRV7-10: validate hub-dr-v7 compliance and hardening evidence
#457 OP-GF-HUBDRV7-11: hub-dr-v7 operational readiness and SLO validation