Installation Manual - 25 Spoke cert-manager, ESO, and Vault auth

How cert-manager, External Secrets Operator, and Vault Kubernetes auth are installed on spoke-dc-v7 through pull GitOps.

This chapter installs the first post-import spoke platform foundation batch:

• cert-manager Operator;
• External Secrets Operator;
• managed-cluster Argo RBAC for ESO bootstrap resources;
• Vault Kubernetes auth for spoke-dc-v7;
• an ESO smoke ExternalSecret.

Do this after the spoke is imported into ACM and the managed Argo CD pull-model application is Synced/Healthy.

Target State

GitOps Files

Add these paths under clusters/spoke-dc-v7:

platform/argocd-extensions/
operators/cert-manager-operator/
operators/external-secrets-operator/
secrets/vault-auth/
secrets/eso/

Include them in clusters/spoke-dc-v7/kustomization.yaml before storage:

resources:
  - platform/argocd-extensions
  - operators/cert-manager-operator
  - operators/external-secrets-operator
  - operators/local-storage-operator
  - operators/odf
  - secrets/vault-auth
  - secrets/eso
  - storage/local-storage
  - storage/odf

The operator subscriptions use the mirrored Red Hat 4.20 catalog source:

source: cs-redhat-operator-index-v4-20
sourceNamespace: openshift-marketplace

Pin the versions used in this deployment:

• cert-manager: cert-manager-operator.v1.19.0;
• ESO: openshift-external-secrets-operator.v1.1.0.

Managed Argo RBAC

The managed spoke Argo controller cannot self-grant cluster-scope permissions. If platform/argocd-extensions is added for the first time, bootstrap the committed ClusterRole and ClusterRoleBinding once:

oc --kubeconfig "$SPOKE_KUBECONFIG" apply \
  -f clusters/spoke-dc-v7/platform/argocd-extensions/clusterrole.yaml \
  -f clusters/spoke-dc-v7/platform/argocd-extensions/clusterrolebinding.yaml

Without this, Argo reports errors like:

cannot patch resource "networkpolicies" in API group "networking.k8s.io"
cannot patch resource "clustersecretstores" in API group "external-secrets.io"

After the one-time bootstrap, Argo owns those RBAC objects from Git.

Configure Vault Auth

Wait until GitOps creates:

external-secrets-operator/vault-token-reviewer-token

Then run the Vault auth script:

export KUBECONFIG="$SPOKE_KUBECONFIG"
export VAULT_ADDR=https://gf-ocp-vault-02.v7.comptech-lab.com:8200
export VAULT_CACERT=/home/ze/codex-opp-agent/secrets/greenfield-vault/tls/ca.crt
export VAULT_TOKEN=<redacted>

/home/ze/ocp-greenfield-deployment/scripts/services/vault/configure-hub-vault-kubernetes-auth.sh \
  --cluster spoke-dc-v7 \
  --kubernetes-host https://api.spoke-dc-v7.v7.comptech-lab.com:6443 \
  --reviewer-namespace external-secrets-operator \
  --reviewer-secret vault-token-reviewer-token \
  --eso-service-account external-secrets-operator-controller-manager \
  --eso-namespace external-secrets-operator

The script configures:

• auth mount kubernetes-spoke-dc-v7;
• policy spoke-dc-v7-eso-secrets;
• role eso-secrets;
• smoke secret path secret/greenfield/openshift/spoke-dc-v7/eso-smoke.

The script must not print reviewer tokens, Vault tokens, or synced secret values.

Issue Found During Install

The original Vault auth script had an invalid jq object expression while rendering the Vault policy. The fixed version is committed in the planning repo:

c1fa1bf Fix Vault Kubernetes auth policy rendering

The policy expression must wrap string concatenation in parentheses:

{
  policy: (
    "path ..." +
    "path ..."
  )
}

Keep this fix before repeating the spoke ESO/Vault-auth gate.

Validation

Validate GitOps:

oc --kubeconfig "$SPOKE_KUBECONFIG" get applications.argoproj.io \
  -n openshift-gitops spoke-dc-v7-cluster-config

Expected:

Synced / Healthy

Validate operators:

oc --kubeconfig "$SPOKE_KUBECONFIG" get csv -n cert-manager-operator
oc --kubeconfig "$SPOKE_KUBECONFIG" get csv -n external-secrets-operator
oc --kubeconfig "$SPOKE_KUBECONFIG" get pods -n cert-manager
oc --kubeconfig "$SPOKE_KUBECONFIG" get pods -n external-secrets

Expected:

• both CSVs are Succeeded;
• cert-manager pods are Running;
• external-secrets pods are Running.

Validate Vault integration:

oc --kubeconfig "$SPOKE_KUBECONFIG" get clustersecretstore vault-platform
oc --kubeconfig "$SPOKE_KUBECONFIG" get externalsecret \
  -n external-secrets-operator eso-smoke
oc --kubeconfig "$SPOKE_KUBECONFIG" get secret \
  -n external-secrets-operator eso-smoke

Expected:

• ClusterSecretStore/vault-platform is Ready=True / Valid;
• ExternalSecret/eso-smoke is Ready=True / SecretSynced;
• Secret external-secrets-operator/eso-smoke exists.

Validate cluster health:

oc --kubeconfig "$SPOKE_KUBECONFIG" get clusterversion version
oc --kubeconfig "$SPOKE_KUBECONFIG" get nodes
oc --kubeconfig "$SPOKE_KUBECONFIG" get co
oc --kubeconfig "$SPOKE_KUBECONFIG" get mcp
oc --kubeconfig "$SPOKE_KUBECONFIG" get storagecluster -n openshift-storage
oc --kubeconfig "$SPOKE_KUBECONFIG" get cephcluster -n openshift-storage

Expected:

• ClusterVersion is available and not progressing;
• all six nodes are Ready;
• no non-steady ClusterOperators;
• master and worker MCPs are updated and not degraded;
• ODF remains Ready;
• Ceph remains HEALTH_OK.

Next Gate

After this chapter passes, spoke-dc-v7 can consume Vault-backed secrets through ESO. The next platform baseline gate is RHACS secured-cluster onboarding for spoke-dc-v7.

Use the same pattern as the hub RHACS install:

• generate the secured-cluster init bundle from the v7 RHACS Central;
• store the init bundle in Vault under the spoke-dc-v7 cluster prefix;
• expose the bundle to the cluster through ESO;
• install only the secured-cluster services on the spoke;
• validate sensor, admission-controller, and collector readiness from both spoke-dc-v7 and RHACS Central.

Do not reuse an old init bundle from another cluster.