Installation Manual - 50 OADP credential and DPA blocker

Why the OADP credential and DPA gate was rolled back and what must be fixed before re-applying it.

This gate attempted to add OADP credential delivery and one DataProtectionApplication on hub-dc-v7 and spoke-dc-v7. The manifests were valid, but live reconciliation was blocked by Vault policy. The change was rolled back, so both clusters are back to the healthy OADP operator-only state.

Governance

Preflight Result

Both clusters were healthy before the attempted apply:

No DPA, BSL, VSL, Schedule, Backup, or Restore existed before the gate.

Intended Shape

The attempted commit added an ExternalSecret and one DPA per cluster.

Credential shape:

Vault path: secret/greenfield/object-storage/minio/users/oadp-backup
Kubernetes Secret: openshift-adp/cloud-credentials
Secret key: cloud

DPA layout:

Common DPA settings:

defaultPlugins: openshift, aws, csi
featureFlags: EnableCSI
nodeAgent: enabled with Kopia
s3ForcePathStyle: "true"
region: default
Velero env: AWS_RETRY_MODE=standard, AWS_MAX_ATTEMPTS=10
Velero memory request: 512Mi

No Schedule, Backup, Restore, BackupStorageLocation, or VolumeSnapshotLocation resources were added directly.

Validation Failure

These checks passed:

oc kustomize clusters/hub-dc-v7
oc kustomize clusters/spoke-dc-v7
oc kustomize clusters/hub-dc-v7/platform/backup/oadp
oc kustomize clusters/spoke-dc-v7/platform/backup/oadp
server-side dry-run for both ExternalSecrets and both DPAs

Live reconciliation then failed:

ExternalSecret/oadp-cloud-credentials: Ready=False, reason=SecretSyncedError
Secret/cloud-credentials: missing
DPA/hub-dc-v7: Reconciled=False, Secret "cloud-credentials" not found

The blocking event was Vault HTTP 403 permission denied for:

secret/data/greenfield/object-storage/minio/users/oadp-backup

The locally available Vault administrator token custody was stale or invalid, so the Vault policy could not be fixed in this gate.

Token Custody Recheck

A follow-up recheck on 2026-05-17T19:15Z confirmed the Vault access blocker was still unresolved:

No Vault policy write, GitOps reapply, Kubernetes Secret creation, DPA creation, schedule creation, or MinIO credential read was performed during the recheck.

Vault VM Admin Token Attempt

A direct Vault VM recheck on 2026-05-17T19:31Z confirmed the VMs are reachable, but VM access alone cannot create a Vault administrator token.

No administrator token was created. Vault administrator-token creation still requires either an existing privileged Vault token or three valid main-Vault Shamir recovery key shares.

Rollback

The attempted commit was reverted:

ac0802fca3c0f1f0357e7c3d64f16bf28410a1ee

Final state after rollback:

Runtime state:

DPA/BSL/VSL/Schedule/Backup/Restore: absent on both clusters
Secret/cloud-credentials: absent on both clusters
OADP operator: still installed and healthy

Required Fix

Use a valid Vault administrator token and grant ESO narrow read access to the OADP MinIO credential path.

Preferred policy:

path "secret/data/greenfield/object-storage/minio/users/oadp-backup" {
  capabilities = ["read"]
}

path "secret/metadata/greenfield/object-storage/minio/users/oadp-backup" {
  capabilities = ["read"]
}

Attach that policy either to the existing hub/spoke Kubernetes auth roles used by ESO, or create a dedicated OADP Vault role and matching ClusterSecretStore.

After Vault access is fixed, re-apply by reverting rollback commit:

git revert ac0802fca3c0f1f0357e7c3d64f16bf28410a1ee

Then validate:

oc -n openshift-adp get externalsecret,secret,dpa,bsl
oc -n openshift-adp get schedule,backup,restore

Do not print Secret/cloud-credentials data. Confirm only that the Secret has the expected key name: cloud.