spoke-dc-v6 — Workload Cluster Topology

Hybrid 3 VM-master + 3 physical-worker workload cluster with ODF, the spoke pull-model Argo CD, ACS Sensor, OADP, Loki/Tempo, Quay, Tekton, and applications.

spoke-dc-v6 is the workload cluster — three VM masters and three physical workers backed by OpenShift Data Foundation. It runs every operator that needs Ceph-backed storage, an object-storage bucket, or workload compute: ODF/LSO, the spoke pull-model Argo CD, RHACS Sensor, OADP, ESO, Loki/Tempo/OpenTelemetry, OSSM/Kiali, Tekton, Quay, and the application tenants.

What’s on the cluster

Cluster shape

VM masters vs physical workers

The split is deliberate:

• VM masters live on the same KVM hypervisor as the hub. They are deterministic-MAC libvirt domains with vTPM and br30 attachment. Master upgrades are quick (snapshot, reboot).
• Physical workers are HPE servers with iLO Redfish-controlled boot, NS204i RAID1 boot volumes (NVMe), and MR408i single-disk data volumes for ODF OSDs. They carry the application workload, ODF Ceph daemons, and the heavy operands (Loki, Tempo, Quay, ScannerV4).

Worker hardware inventory:

Total data device capacity: 20 disks. After issue #117’s destructive cleanup, all 20 were clean by wipefs -n.

Failure-domain labels

Workers are labeled with rack + zone so ODF can place OSDs / MONs across failure domains:

oc --kubeconfig "$K_SPOKE" label node spoke-dc-v6-worker-0 \
  topology.kubernetes.io/rack=gold-1 topology.kubernetes.io/zone=dc-lab --overwrite
oc --kubeconfig "$K_SPOKE" label node spoke-dc-v6-worker-1 \
  topology.kubernetes.io/rack=gold-2 topology.kubernetes.io/zone=dc-lab --overwrite
oc --kubeconfig "$K_SPOKE" label node spoke-dc-v6-worker-2 \
  topology.kubernetes.io/rack=gpu-01 topology.kubernetes.io/zone=dc-lab --overwrite

ODF StorageCluster.spec.placement (covered in §8 storage / ODF on spoke) uses rack as the failure domain — one OSD per rack, MON quorum spread across all three racks.

Storage stack — LSO + ODF

ADR 0004 says storage lives on the spoke. The stack is:

1. Local Storage Operator (local-storage-operator.v4.20.0-202604140241) installed in openshift-local-storage. It runs a LocalVolumeSet that discovers MR408i devices via path/size filters (excluding NS204i boot disks).
2. OpenShift Data Foundation (odf-operator.v4.20.10-rhodf) installed in openshift-storage with the full dependency operator set (CephCSI, MCG/NooBaa, ODF dependencies, Rook). The StorageCluster consumes the LSO-provided PVs as OSDs.
3. StorageClasses exposed cluster-wide:
   • ocs-storagecluster-ceph-rbd — block, RWO (default for new workload PVCs)
   • ocs-storagecluster-cephfs — filesystem, RWX
   • ocs-storagecluster-ceph-rgw — S3-compatible object via RGW
4. NooBaa (MCG) — Multi-Cloud Gateway, exposed via the objectbucket.io API. Tenants request buckets through ObjectBucketClaim resources; the OBC controller materializes an S3 bucket plus an AWS_*_KEY_* Secret.

The NooBaa OBC → operand storage-Secret bridge (for Loki, Tempo, Quay) uses External Secrets Operator’s kubernetes-provider with a template to rewrite the OBC-emitted Secret keys into the lowercase variants the LokiStack and TempoStack expect. Pattern lives at clusters/spoke-dc-v6/platform-services/tracing/externalsecret-tempo-storage.yaml.

Spoke pull-model Argo CD

The spoke runs OpenShift GitOps differently from the hub. The ACM gitops-addon (enabled via GitOpsCluster.spec.gitopsAddon.enabled: true on the hub) installs OpenShift GitOps on the spoke as a Deployment, not as an OLM CSV. The namespace on the spoke is openshift-gitops, but the controller is named acm-openshift-gitops-argocd-application-controller (per the addon’s naming convention).

This is intentional — Red Hat’s reference pull-model installs the spoke Argo via the addon Deployment so the hub can manage its lifecycle through ACM. It is not drift; the memory feedback_acm_gitops_pull_pattern.md explicitly notes this.

The spoke Argo CD’s serviceaccount: acm-openshift-gitops-argocd-application-controller has a least-privilege ClusterRole that grants get/list/watch on everything plus full verbs on a fixed allowlist of API groups. Anything outside the allowlist hits forbidden: cannot patch .... The single home for RBAC extensions is clusters/spoke-dc-v6/platform/argocd-extensions/clusterrole.yaml — see §5.7 spoke extension RBAC for the full list.

Installed operators (as of 2026-05-11)

The Subscriptions live in clusters/spoke-dc-v6/operators/* (per operator).

Workload tenants

Per ADR 0024, tenant boundary registration lives in the platform repo under clusters/spoke-dc-v6/tenants/<tenant>/ — namespace, AppProject scope, quotas, baseline NetworkPolicies, base ServiceAccounts. Per-release Deployment/Route/Service manifests belong in the tenant’s *-gitops repo, not in the platform repo.

The current tenant directory apps-platform-sample/ is the placeholder tenant used for testing the Open Liberty operator path and the Tekton pipeline templates. Future tenants (Open Liberty applications, JBoss/EAP services, Spring Boot APIs, Node.js APIs) onboard through the same _template/ pattern.

Health snapshot commands

K_SPOKE=/home/ze/.kube/configs/spoke-dc-v6.kubeconfig

# Cluster baseline
oc --kubeconfig "$K_SPOKE" get nodes -o wide
oc --kubeconfig "$K_SPOKE" get clusterversion version
oc --kubeconfig "$K_SPOKE" get co | awk \
  'NR==1 || $3 != "True" || $4 != "False" || $5 != "False"'

# ODF + LSO
oc --kubeconfig "$K_SPOKE" -n openshift-storage get storagecluster,noobaa
oc --kubeconfig "$K_SPOKE" -n openshift-storage get cephcluster \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.ceph.health}{"\n"}{end}'
oc --kubeconfig "$K_SPOKE" get storageclass | egrep 'ocs-storagecluster|NAME'

# Spoke Argo
oc --kubeconfig "$K_SPOKE" -n openshift-gitops get applications.argoproj.io -o wide
oc --kubeconfig "$K_SPOKE" -n openshift-gitops get app spoke-dc-v6-cluster-config \
  -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{" "}{.status.sync.revision}{"\n"}'

# Operator sets
oc --kubeconfig "$K_SPOKE" get csv -A | awk \
  '/Succeeded/ {print $1, $2}' | sort -u

Expected steady state:

• All 6 nodes Ready.
• ClusterVersion 4.20.18, Available=True, Progressing=False.
• ODF StorageCluster Ready; Ceph HEALTH_OK; NooBaa Ready; storage classes present.
• spoke-dc-v6-cluster-config Application Synced Healthy.
• All listed operator CSVs Succeeded.

References

• connection-details/openshift-spoke-dc-v6.md — endpoints, node inventory, install notes (including physical-worker MAC correction).
• connection-details/platform-admin-handoff.md — installed-operator baseline, ODF expected state, validation commands.
• plans/disconnected-rebuild/environments/dc-lab/spoke-dc-v6/README.md, physical-worker-boot-plan.md, validation.md.
• plans/disconnected-rebuild/environments/dc-lab/odf-storage-inventory.md.
• adr/0004-management-only-hubs.md, adr/0018-acm-openshift-gitops-pull-model-v6.md, adr/0019-nexus-only-image-supply-chain.md, adr/0024-openshift-only-platform-gitops-boundary.md.
• platform-gitops clusters/spoke-dc-v6/ tree (operators, storage, security, platform-services, tenants).