Installation Manual - 74 Vault replacement phase closeout

Final read-only closeout for the v7 Vault replacement phase after R1 cutover, backup proof, old VM deletion, and stale DNS cleanup.

This chapter records the final read-only closeout for the greenfield v7 Vault replacement phase.

The phase replaced a lost-custody Vault deployment with a new R1 Vault path, migrated OpenShift consumers, proved backup and restore behavior, removed the old VM and disk footprint, and cleaned up the final stale DNS record.

Governance

FieldValue
IssueOP-GF-VAULTRECOVERY-1 / #389
MilestoneWorkspace Governance
ADRADR 0028: Greenfield Vault Replacement After Custody Loss
Existing controlsADR 0016 and ADR 0025

Final State

Replacement R1 Vault remains the active Vault path for greenfield OpenShift consumers.

AreaFinal state
R1 Vault VMsrunning, autostart enabled
stable Vault DNSvault.v7.comptech-lab.com -> 30.30.200.35-.37
old Vault VM definitionsabsent
old Vault disk imagesabsent
old Vault node DNS recordsabsent
PowerDNS serial45
GitOps revision0bb0cca
OADP schedulesnormal daily schedules restored

Final Read-Only Validation

Libvirt validation from dl385-2:

Old VMDomainDisk image
gf-ocp-vault-seed-01absentabsent
gf-ocp-vault-01absentabsent
gf-ocp-vault-02absentabsent
gf-ocp-vault-03absentabsent

R1 VM state:

R1 VMStateAutostart
gf-ocp-vault-r1-seed-01runningenabled
gf-ocp-vault-r1-01runningenabled
gf-ocp-vault-r1-02runningenabled
gf-ocp-vault-r1-03runningenabled

Vault health:

Endpoint setResult
old direct IPs 30.30.200.30-.33HTTP 000
R1 direct IPs 30.30.200.35-.37HTTP 200

DNS validation:

NameResult
vault.v7.comptech-lab.com30.30.200.35, 30.30.200.36, 30.30.200.37
gf-ocp-vault-seed-01.v7.comptech-lab.comNO_RECORD
gf-ocp-vault-01.v7.comptech-lab.comNO_RECORD
gf-ocp-vault-02.v7.comptech-lab.comNO_RECORD
gf-ocp-vault-03.v7.comptech-lab.comNO_RECORD

PowerDNS authoritative zone contains only the stable Vault R1 records for Vault access and reports SOA serial 45.

OpenShift Validation

Cluster health:

ClusterOpenShiftNodesClusterOperators
hub-dc-v74.20.183/3 Readysteady
spoke-dc-v74.20.186/6 Readysteady

Argo CD:

ApplicationSyncHealthRevision
hub-dc-v7-bootstrapSyncedHealthy0bb0cca
spoke-dc-v7-cluster-configSyncedHealthy0bb0cca

External Secrets:

ClusterResult
hub6/6 ExternalSecrets Ready
spoke6/6 ExternalSecrets Ready

R1-backed ClusterSecretStores remained Ready/Valid on both clusters:

  • vault-r1-eso-smoke;
  • vault-r1-oadp;
  • vault-r1-rhacs.

Vault egress policies allow only R1 Vault CIDRs:

30.30.200.35/32
30.30.200.36/32
30.30.200.37/32

OADP:

ClusterDPABSLScheduleLatest backupPhaseItemsWarningsErrors
hub-dc-v7ReconciledAvailable15 2 * * *platform-resource-daily-20260518063347Completed10122/1012200
spoke-dc-v7ReconciledAvailable45 2 * * *platform-resource-daily-20260518063423Completed16808/1680800

RHACS:

  • hub Central reported Available;
  • no non-running StackRox pods were found on hub or spoke.

Phase Summary

The completed Vault replacement phase included:

  • accepted ADR and GitHub issue governance;
  • replacement R1 VM allocation;
  • R1 Vault build and custody creation;
  • Kubernetes auth and smoke read validation for hub/spoke;
  • dedicated OADP, ESO smoke, and RHACS replacement stores;
  • MinIO OADP credential rotation and seeding into R1 Vault;
  • OADP DPA validation;
  • ad hoc backup validation;
  • scheduled backup enablement and proof;
  • restore drill;
  • stable Vault DNS promotion;
  • removal of the unused vault-platform store;
  • post-cleanup scheduled backup proof;
  • old Vault retirement readiness;
  • removal of old Vault egress CIDRs;
  • old VM power-off;
  • cold retention soak;
  • post-power-off backup proof;
  • deletion of old VM definitions and disk images;
  • stale old seed DNS cleanup.

Result

The greenfield v7 Vault replacement phase is complete.

The active OpenShift consumers use replacement R1 Vault paths, backup and restore behavior has been proven, and the old lost-custody Vault VM, disk, and DNS footprint is removed from active infrastructure.

Issue #389 was closed after the final read-only validation passed.