GitOps Operating Model — Architecture
The ACM + OpenShift GitOps Basic pull model in detail: hub Argo + spoke Argo + ApplicationSet + ManifestWork, what initiates what, and why pull beats hub-render at fleet scale.
The fleet’s operating model is ACM + OpenShift GitOps Basic pull (per ADR 0018). Both clusters run an Argo CD instance that reconciles its own desired state from the same GitLab repository, with the hub Argo coordinating placement and ApplicationSet generation but never directly applying spoke manifests. The spoke is its own reconciler. This page is the architecture in one diagram and one set of definitions.
The architecture
The defining property: every dashed-green arrow is initiated from the inside out. The klusterlet on the spoke calls the hub. The spoke Argo CD calls GitLab. The hub Argo CD calls GitLab. Nothing on the hub initiates a connection to the spoke API server. This is what makes the pattern survive NAT, firewalls, transient WAN, and (in larger fleets) air-gapped sites between sync windows.
The two Argo CD instances
| Instance | Where | Install | Privilege | Reconciles |
|---|---|---|---|---|
| Hub Argo CD | hub-dc-v6 openshift-gitops namespace | OLM CSV openshift-gitops-operator.v1.20.3 | cluster-admin ClusterRoleBinding | clusters/hub-dc-v6/ (against kubernetes.default.svc) plus the per-spoke Application objects (which it does not apply) |
| Spoke Argo CD | spoke-dc-v6 openshift-gitops namespace | Deployment installed by ACM gitops-addon | Least-privilege ClusterRole argocd-platform-extensions | clusters/spoke-dc-v6/ (against kubernetes.default.svc) |
The hub uses a bound cluster-admin because it has to install ACM, MCE, all platform operators, RHACS Central, and write cluster-scoped resources — a least-privilege ClusterRole would be a fast-moving target as new operators land. The compensating controls are: the hub Argo’s AppProject/hub-dc-v6-platform sourceRepos allowlist restricts it to the LAN GitLab URL only; CODEOWNERS in GitLab restricts who can merge; argocd-platform-extensions doesn’t exist on the hub because it isn’t needed.
The spoke uses a least-privilege ClusterRole because it is exposed by ACM’s pull pattern. The addon-installed Argo’s service account acm-openshift-gitops-argocd-application-controller has get/list/watch on everything plus full verbs on a fixed allowlist of API groups (see §5.5 spoke extension RBAC).
The Application objects, in three roles
The same Argo Application CR plays three different roles depending on where it lives and what labels it carries:
Role 1 — hub bootstrap Application
hub-dc-v6-bootstrap lives in the hub Argo’s openshift-gitops namespace and reconciles clusters/hub-dc-v6/ against kubernetes.default.svc (i.e., the hub itself).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: hub-dc-v6-bootstrap
namespace: openshift-gitops
spec:
project: hub-dc-v6-platform
source:
repoURL: http://<gitlab-vm>/comptech-platform/openshift-ops/openshift-platform-gitops.git
targetRevision: main
path: clusters/hub-dc-v6
destination:
server: https://kubernetes.default.svc
namespace: openshift-gitops
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
This is the only object an operator has to oc apply by hand (after the OpenShift GitOps operator is installed). From there everything else flows through GitOps.
Role 2 — ApplicationSet-generated “Application as ManifestWork wrapper”
The hub Argo’s spoke-cluster-config-pull ApplicationSet generates one Application per spoke. Critical features of the generated Application:
metadata.labels.apps.open-cluster-management.io/pull-to-ocm-managed-cluster: "true"— tells ACM to wrap it as ManifestWork.metadata.annotations.argocd.argoproj.io/skip-reconcile: "true"— tells the hub Argo CD to not reconcile it. The hub Argo is the owner-of-record only; ACM ships it to the spoke.metadata.annotations.apps.open-cluster-management.io/ocm-managed-cluster: "{{name}}"— tells ACM which spoke to target.spec.project: default— lands on the spoke, wherehub-dc-v6-platformAppProject doesn’t exist;defaultdoes.spec.destination.server: https://kubernetes.default.svc— points at the spoke’s local API once it lands there.
Role 3 — spoke-side Application
When the ManifestWork lands on the spoke, the wrapped Application is materialized in the spoke’s openshift-gitops namespace. The spoke Argo CD finds it and reconciles it normally — pulling clusters/spoke-dc-v6/ from GitLab and applying to the local API.
Net effect: from the spoke Argo’s perspective, it has a single Application called spoke-dc-v6-cluster-config that reconciles clusters/spoke-dc-v6/. From the hub’s perspective, the same Application was generated by an ApplicationSet and shipped via ManifestWork.
What initiates what
| Connection | Initiated by | Direction |
|---|---|---|
klusterlet → hub registration controller | klusterlet (spoke) | outbound from spoke |
klusterlet work agent → ACM ManifestWork pull | klusterlet (spoke) | outbound from spoke |
Hub Argo → GitLab clusters/hub-dc-v6/ | hub Argo | outbound from hub |
Spoke Argo → GitLab clusters/spoke-dc-v6/ | spoke Argo | outbound from spoke |
| Spoke Argo → spoke API server | spoke Argo | local (in-cluster) |
| Hub Argo → hub API server | hub Argo | local (in-cluster) |
| Hub Argo → spoke API server | never | n/a |
The last row is the whole point. Without it, the hub doesn’t need spoke kubeconfigs, doesn’t need to reach the spoke’s API, doesn’t need a hole punched through a NAT for spoke API access.
Sync-wave ordering, in summary
OpenShift GitOps respects argocd.argoproj.io/sync-wave annotations. The convention used across platform-gitops:
| Wave | Role |
|---|---|
| 0 | RBAC scaffolding (ClusterRoles, ClusterRoleBindings, hub Argo cluster-admin binding, spoke argocd-platform-extensions, the AppProject manager Role for the bootstrap namespace) |
| 5 | Namespace, OperatorGroup |
| 10 | Subscription (operator install) |
| 20 | Operator CR (MultiClusterHub, ClusterCatalog, IDMS/ITMS, StorageCluster, LokiStack, …) |
| 30 | First-class operand objects (ManagedCluster, KlusterletAddonConfig, ManagedClusterSetBinding) |
| 40 | Per-spoke addon config |
| 50 | Placement |
| 60 | GitOpsCluster |
| 70 | ApplicationSet |
Details and the full per-resource table in §5.3 sync-wave conventions.
What this isn’t
This model is not the Argo CD Agent (now Argo CD ApplicationSet pull controller in newer versions). Argo CD Agent installs an “agent” Argo on the spoke that talks to a “principal” Argo on the hub via a dedicated bidirectional channel, and disables the hub Argo’s normal application controller in the process. ADR 0018 explicitly rejects this for the v6 baseline:
Do not use Argo CD Agent for this baseline. It may be reconsidered later only through a separate accepted ADR and after current Red Hat support, maturity, and operational load implications are reviewed.
The two distinctions:
| ACM + OpenShift GitOps Basic pull (this fleet) | Argo CD Agent | |
|---|---|---|
| Spoke Argo install | ACM gitops-addon Deployment | Argo CD Agent install |
| ManifestWork dispatch | Yes — ACM’s app-lifecycle pillar wraps Apps | No — Agent has its own dispatch channel |
| Hub Argo behavior | Normal — owns hub-side AppSet, doesn’t reconcile spoke Apps | Demoted to “principal”, doesn’t run application-controller for spoke targets |
| Cluster registration | ACM ManagedCluster | Argo CD Agent registration |
| Maturity in OpenShift | Stable since OCP 4.16 | Newer, less Red Hat-tested |
We chose Basic pull because it leans on ACM components that are already needed for cluster lifecycle, governance, and the multicluster console.
References
adr/0003-gitops-basic-pull-model.md— pre-v6 origin of the pull model.adr/0018-acm-openshift-gitops-pull-model-v6.md— current decision for the v6 fleet.connection-details/openshift-gitops-hub-dc-v6.md— hub Argo CD instance + bootstrap Application.connection-details/platform-admin-handoff.md— operating workflow summary.- platform-gitops
clusters/hub-dc-v6/gitops-control/— the hub-side glue (AppProject, ApplicationSet, cluster-admin binding).