ADR 0018 — ACM + OpenShift GitOps pull model for v6

Hub coordinates ACM placement and ApplicationSet propagation; each managed workload cluster runs OpenShift GitOps locally and reconciles from internal GitLab. Supersedes ADR 0003.

Date: 2026-05-09 Status: Accepted. Supersedes ADR 0003 (historical). Constrains ADR 0015 on OpenShift platform operations.

Context

The v6 rebuild has a healthy hub-dc-v6 compact management cluster and a healthy spoke-dc-v6 workload cluster. At the time this ADR was accepted, hub-dc-v6 ran only a bootstrap OpenShift GitOps Application from internal GitLab. It did not yet define ACM, spoke registration, management ApplicationSets, ODF for the workload cluster, or workload-cluster overlays. The ADR’s job is to fix the shape of that work before it starts.

Previous hub-heavy GitOps operation put too much pressure on the management cluster. The older hub-dc / spoke-dc design already used the Red Hat Advanced Cluster Management + OpenShift GitOps Basic pull pattern, where managed clusters reconcile local Argo CD Applications. That lesson is still valid for the v6 rebuild.

The operating requirements:

  • GitLab is the operational source of truth for platform and application delivery.
  • GitHub opp-full-plat remains planning / ADR / issue / milestone / handoff guidance only.
  • Normal OpenShift changes must be GitOps-driven.
  • oc, console, SSH, direct API mutation are break-glass paths only (see ADR 0025).
  • Application teams must not have write access to the OpenShift platform operations repository.
  • spoke-dc-v6 ODF / LSO must not be recreated manually.

Decision

Use ACM + OpenShift GitOps Basic pull model as the v6 baseline for workload-cluster management.

Hub responsibilities

hub-dc-v6 owns management coordination only:

  • ACM/MCE installation and configuration.
  • ManagedCluster, ManagedClusterSet, and placement policy.
  • GitOpsCluster wiring between ACM placement decisions and OpenShift GitOps.
  • Hub-local OpenShift GitOps bootstrap and hub-local platform resources.
  • ApplicationSet definitions that generate lightweight managed-cluster application objects.

hub-dc-v6 must not directly push routine spoke platform configuration, operator installs, storage resources, or application workloads into spoke-dc-v6. The hub creates an Application (via ApplicationSet) targeted at the spoke; the spoke’s local Argo CD reconciles it.

Spoke responsibilities

Each managed workload cluster runs OpenShift GitOps locally. For spoke-dc-v6, the local Argo CD pulls from the internal GitLab platform repository and reconciles local desired state with:

destination:
  server: https://kubernetes.default.svc

Managed workload applications must use the Red Hat pull-model annotations and ACM propagation flow. The hub-side Application is a coordination object; the spoke-local Argo CD instance is the reconciler.

Do not use Argo CD Agent for this baseline. It may be reconsidered later only through a separate accepted ADR and after current Red Hat support, maturity, and operational load implications are reviewed.

Repository baseline

The active platform GitOps repository is:

http://<internal-gitlab-host>/comptech-platform/openshift-ops/openshift-platform-gitops.git

The local operator clone is /home/ze/platform-gitops (per the platform-gitops clone-path convention).

Expected high-level shape:

openshift-platform-gitops/
  clusters/
    hub-dc-v6/
      bootstrap/
      platform/
    spoke-dc-v6/
      cluster-config/
      operators/
      storage/

  components/
    acm/
    gitops/
    odf/
    lso/
    appprojects/
    tenants/

  appsets/
    managed-cluster-config/
    managed-cluster-operators/

Historical repositories and old spoke-dc manifests are reference material only. They must not be copied directly into v6 without updating topology, operator channels, versions, placements, credentials, and ownership metadata.

Sequencing

Use this order before returning to spoke-dc-v6 ODF:

  1. Keep hub-dc-v6 bootstrap GitOps healthy.
  2. Add ACM/MCE and required hub management namespaces through GitOps.
  3. Register or import spoke-dc-v6 into ACM.
  4. Install or enable OpenShift GitOps on spoke-dc-v6 through tracked management GitOps, not console drift.
  5. Seed required internal GitLab credentials by reference only; do not commit credentials.
  6. Create pull-model ApplicationSets for managed-cluster configuration.
  7. Validate placement decisions, ManifestWork propagation, spoke-local Argo CD Applications, and MulticlusterApplicationSetReport status.
  8. Add LSO and ODF desired state for spoke-dc-v6.
  9. Let spoke-local Argo CD install LSO/ODF and reconcile storage resources.

Alternatives considered

Hub-push GitOps (Argo CD on the hub reconciles spoke clusters directly). The pre-v6 baseline for some shorter periods. Rejected because:

  • Every spoke change goes through the hub’s Argo CD; the hub repoServer becomes the throughput bottleneck.
  • Spoke clusters can’t recover their own state when the hub is unhealthy.
  • ACM placement and GitOps don’t compose as cleanly — ACM has ManifestWork for the same purpose, and the hub’s Argo CD ends up doing duplicate work.

Argo CD Agent (Argo CD’s experimental “agent” mode). Attractive because it formalizes the hub/spoke split. Rejected for the v6 baseline because:

  • Red Hat support for Argo CD Agent in OpenShift GitOps was not at the maturity the lab needs at the time of acceptance.
  • The lab wants a single pattern across hubs and spokes (Argo CD instance on each), not a separate “agent” runtime.
  • May be revisited in a future ADR.

No GitOps; do everything via oc apply from CI. Considered pre-v6 and rejected for the reasons in ADR 0015 and ADR 0025. Push-mode operations leave the cluster unable to reconcile its own state.

Use Argo CD only on the hub; use ACM Subscription / Channel for application delivery. Older pattern. Rejected because Argo CD’s ApplicationSet + the ACM Placement-Decision generator gives a cleaner declarative path than Subscription/Channel.

Guardrails and gotchas

  • Placement must select only the intended active workload clusters. Don’t let unreachable DR clusters or standby hubs match active placement labels.
  • An unavailable managed cluster can receive an ACM unavailable taint, leave placement decisions, and trigger unexpected ApplicationSet / ManifestWork pruning. (See the routes-CRD incident page for a related failure mode.) Use explicit labels and separate placements for active, standby, and future clusters.
  • Every managed cluster must have network access to internal GitLab and any registry endpoints it needs.
  • Repository credentials must be local secrets or managed by an approved secret manager; Git stores references only.
  • Operator install resources, namespaces, StorageClasses, LocalVolumeSets, StorageClusters, AppProjects, tenant boundaries, and workload registrations must be represented in Git before they are applied.
  • Break-glass changes require a tracked issue, reason, expiry or rollback plan, and follow-up Git backport (see ADR 0025).

Consequences

  • More moving parts than a hub-push model, but reconciliation load distributes to managed clusters and hub-dc-v6 does not become the executor for every spoke workload change.
  • Each managed cluster must run and operate its own OpenShift GitOps instance. This adds per-cluster bootstrap and credential-management work, but it keeps spoke operations available even when the hub is only coordinating status and placement.
  • ODF/LSO on spoke-dc-v6 remains blocked until this management GitOps baseline exists and the relevant desired-state overlays are reviewed.
  • Supersedes ADR 0003 for the v6 rebuild. ADR 0003 remains useful as historical context for the previous hub-dc / spoke-dc pull-model implementation.
  • Constrains ADR 0015: the federated repository architecture remains accepted, but OpenShift platform operations inside that architecture must use this ACM + OpenShift GitOps pull model unless a future ADR changes the cluster-management pattern.

References

  • Source: opp-full-plat/adr/0018-acm-openshift-gitops-pull-model-v6.md
  • Superseded predecessor: opp-full-plat/adr/0003-gitops-basic-pull-model.md
  • Federation context: ADR 0015
  • Nexus-only supply chain (binds the same v6 platform repo): ADR 0019
  • GitOps-only operations rule: opp-full-plat/adr/0025-gitops-only-operations-break-glass.md
  • platform-gitops clone path: operator-memory reference_platform_gitops_mr_path.md
  • ACM routes-CRD incident: operator-memory project_acm_gitops_addon_routes_crd.md
  • RHACM background: the RHACM blog post

Last reviewed: 2026-05-11