Installation Manual - 68 Old Vault stage 1 early backup validation

Temporary OADP schedule acceleration and validation after old Vault stage 1 retirement cleanup.

This chapter records the post-stage-1 backup validation after old main Vault DNS and External Secrets egress references were removed.

The schedules were temporarily accelerated, the scheduled backups completed, and the normal daily schedule was restored in GitOps.

Governance

FieldValue
IssueOP-GF-VAULTRECOVERY-1 / #389
MilestoneWorkspace Governance
ADRADR 0028: Greenfield Vault Replacement After Custody Loss
Existing controlsADR 0016 and ADR 0025

Temporary Schedule Acceleration

Temporary GitOps commit:

3f7f357 Temporarily accelerate OADP schedules

Temporary schedule values:

ClusterTemporary schedule
hub-dc-v717 1 * * *
spoke-dc-v718 1 * * *

Validation before live reconciliation:

  • local oc kustomize render passed for hub and spoke;
  • git diff --check passed;
  • server-side dry-run accepted both rendered overlays.

Argo CD converged to temporary commit 3f7f357419a8088731cee48cb0229099e7d9b46e, and live schedules matched the temporary values before the backup windows.

Early Backup Results

ClusterBackupResult
hub-dc-v7platform-resource-daily-20260518011746Completed, 10432/10432, warnings 0, errors 0
spoke-dc-v7platform-resource-daily-20260518011823Completed, 15877/15877, warnings 0, errors 0

The first watch query was discarded because it used a brittle epoch conversion and matched older backup CRs. The accepted watch used ISO timestamp filtering for backups created after 2026-05-18T01:16:00Z.

Normal Schedule Restore

Restore GitOps commit:

c138c6c Restore OADP daily schedules

Restored schedule values:

ClusterRestored scheduleLast backup
hub-dc-v715 2 * * *2026-05-18T01:17:46Z
spoke-dc-v745 2 * * *2026-05-18T01:18:23Z

Server-side dry-run accepted both restore overlays before Argo reconciliation.

Final Argo CD state:

Cluster contextApplicationSyncHealthRevision
hubhub-dc-v7-bootstrapSyncedHealthyc138c6cebba560de4298c4189b7028fef50c7f02
hubspoke-dc-v7-cluster-configSyncedHealthyc138c6cebba560de4298c4189b7028fef50c7f02
spokespoke-dc-v7-cluster-configSyncedHealthyc138c6cebba560de4298c4189b7028fef50c7f02

Post-Backup Health Snapshot

OADP:

ClusterDPABSLSchedule
hub-dc-v7ReconciledAvailable15 2 * * *
spoke-dc-v7ReconciledAvailable45 2 * * *

External Secrets:

ClusterTotal ExternalSecretsReady ExternalSecrets
hub-dc-v766
spoke-dc-v766

Vault egress policies still allow only R1 Vault CIDRs:

30.30.200.35/32,30.30.200.36/32,30.30.200.37/32

ClusterOperators:

  • no hub exceptions;
  • no spoke exceptions.

RHACS:

  • hub Central remains Available;
  • no non-running StackRox pods were reported on hub or spoke.

Vault R1 health:

EndpointHealth HTTP code
30.30.200.35:8200200
30.30.200.36:8200200
30.30.200.37:8200200

Stage 1 Cleanup Preservation Check

DNS state through the private resolver:

NameResult
gf-ocp-vault-01.v7.comptech-lab.comno A record
gf-ocp-vault-02.v7.comptech-lab.comno A record
gf-ocp-vault-03.v7.comptech-lab.comno A record
gf-ocp-vault-seed-01.v7.comptech-lab.com30.30.200.30
vault.v7.comptech-lab.com30.30.200.35, 30.30.200.36, 30.30.200.37

PowerDNS zone serial remained:

44

All old and replacement Vault VMs remained running on dl385-2.

Old Vault direct health still returned HTTP 200 for:

  • 30.30.200.30
  • 30.30.200.31
  • 30.30.200.32
  • 30.30.200.33

No Vault VM was stopped and no disk image was deleted.

Result

The post-stage-1 early scheduled backup validation passed.

The next governed gate can decide whether to power off the old Vault VMs. Disk deletion remains a separate final retention decision.