Installation Manual - 16 Hub External Secrets and Vault auth

How External Secrets Operator is installed on the hub and wired to Vault using Kubernetes auth.

This chapter installs External Secrets Operator and configures Vault Kubernetes auth for hub-dc-v7.

Use this gate before ACM/MCE, RHACS, or any hub service that needs secrets from Vault.

Target State

GitOps Files

Add:

clusters/hub-dc-v7/operators/external-secrets-operator/
clusters/hub-dc-v7/secrets/vault-auth/
clusters/hub-dc-v7/secrets/eso/

Then include these paths in:

clusters/hub-dc-v7/kustomization.yaml

The operator layer includes:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-external-secrets-operator
  namespace: external-secrets-operator
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-external-secrets-operator
  source: cs-redhat-operator-index-v4-20
  sourceNamespace: openshift-marketplace
  startingCSV: openshift-external-secrets-operator.v1.1.0

The ExternalSecretsConfig is named cluster:

apiVersion: operator.openshift.io/v1alpha1
kind: ExternalSecretsConfig
metadata:
  name: cluster
spec: {}

Vault Auth Service Account

Create the reviewer service account and token in external-secrets-operator:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-token-reviewer
  namespace: external-secrets-operator
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token-reviewer-token
  namespace: external-secrets-operator
  annotations:
    kubernetes.io/service-account.name: vault-token-reviewer
type: kubernetes.io/service-account-token

Bind the service account to system:auth-delegator so Vault can review tokens.

Vault Egress NetworkPolicy

The Red Hat External Secrets operand creates a default-deny policy in the external-secrets namespace. It allows DNS and API server access, but not Vault. Add an explicit egress policy for the controller:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-vault-vms
  namespace: external-secrets
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: external-secrets
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 30.30.200.31/32
        - ipBlock:
            cidr: 30.30.200.32/32
        - ipBlock:
            cidr: 30.30.200.33/32
      ports:
        - port: 8200
          protocol: TCP

Without this policy, the ClusterSecretStore may report:

unable to log in with Kubernetes auth: context deadline exceeded

ClusterSecretStore

Create the cluster-wide store:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-platform
spec:
  provider:
    vault:
      server: https://vault.v7.comptech-lab.com:8200
      path: secret
      version: v2
      caBundle: <base64-vault-ca>
      auth:
        kubernetes:
          mountPath: kubernetes-hub-dc-v7
          role: eso-secrets
          serviceAccountRef:
            name: external-secrets-operator-controller-manager
            namespace: external-secrets-operator
            audiences:
              - vault

Use the greenfield Vault CA bundle. Do not use insecureSkipVerify.

Configure Vault

Run the repeatable script after the GitOps sync has created vault-token-reviewer-token:

export KUBECONFIG=/home/ze/ocp-greenfield-deployment/artifacts/openshift/hub-dc-v7/auth/kubeconfig
export VAULT_ADDR=https://gf-ocp-vault-02.v7.comptech-lab.com:8200
export VAULT_CACERT=/home/ze/codex-opp-agent/secrets/greenfield-vault/tls/ca.crt
export VAULT_TOKEN=<redacted>

/home/ze/ocp-greenfield-deployment/scripts/services/vault/configure-hub-vault-kubernetes-auth.sh \
  --cluster hub-dc-v7 \
  --kubernetes-host https://api.hub-dc-v7.v7.comptech-lab.com:6443 \
  --reviewer-namespace external-secrets-operator \
  --reviewer-secret vault-token-reviewer-token \
  --eso-service-account external-secrets-operator-controller-manager \
  --eso-namespace external-secrets-operator

The script configures:

• Vault auth mount kubernetes-hub-dc-v7;
• Vault policy hub-dc-v7-eso-secrets;
• Vault role eso-secrets;
• smoke secret path secret/greenfield/openshift/hub-dc-v7/eso-smoke.

The script does not print reviewer tokens, Vault tokens, or secret values.

Smoke ExternalSecret

Add a smoke ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: eso-smoke
  namespace: external-secrets-operator
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-platform
  target:
    name: eso-smoke
    creationPolicy: Owner
    deletionPolicy: Retain
  data:
    - secretKey: hello
      remoteRef:
        key: greenfield/openshift/hub-dc-v7/eso-smoke
        property: hello
        conversionStrategy: Default
        decodingStrategy: None
        metadataPolicy: None

Keep the defaulted fields explicit so Argo does not report persistent drift.

Validation

Validate the operator:

oc get sub -n external-secrets-operator
oc get csv -n external-secrets-operator
oc get pods -n external-secrets-operator
oc get pods -n external-secrets

Expected state:

• subscription uses stable-v1;
• openshift-external-secrets-operator.v1.1.0 is Succeeded;
• operator controller pod is Running;
• external-secrets, cert-controller, and webhook pods are Running.

Validate Vault wiring:

oc get clustersecretstore vault-platform
oc -n external-secrets-operator get externalsecret eso-smoke
oc -n external-secrets-operator get secret eso-smoke

Expected state:

• ClusterSecretStore condition Ready=True;
• smoke ExternalSecret condition Ready=True;
• target Secret eso-smoke exists.

Validate Argo:

oc -n openshift-gitops get application.argoproj.io hub-dc-v7-bootstrap \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status,REV:.status.sync.revision

Expected state:

hub-dc-v7-bootstrap   Synced   Healthy

Completion State

After this chapter, the hub has a working cluster-level Vault secret store. Future platform components can consume secrets through GitOps-managed ExternalSecret resources instead of inline Kubernetes Secrets.

The next dependency gate should install ACM/MCE on hub-dc-v7.