GitOps consume — overview

How the platform-managed ApplicationSet discovers tenant overlays in GitLab and turns them into namespace-scoped Argo Applications on each spoke.

This subsection is for platform-admin operators, not tenant developers. It describes how a tenant overlay sitting in a GitLab app repo turns into a running Deployment on spoke-dc-v6. The pipe between Git and the cluster is owned by platform-gitops; tenants do not write Application or ApplicationSet YAML, and the platform team does not write tenant Deployment YAML.

Whiteboard — Tenant overlay to spoke namespace

Read the diagram top-down:

1. The tenant app repo in GitLab carries the overlay layout from the app-repo overlay contract.
2. The platform-managed ApplicationSet in platform-gitops/clusters/spoke-dc-v6/appsets/ watches that repo with a git generator. For every directory matching apps/*/*/overlays/*/kustomization.yaml, it templates one Argo Application per (team, app, env, target-cluster) tuple.
3. The ACM Placement / PlacementDecision pair feeds the ApplicationSet’s clusterDecisionResource generator, so an overlay only becomes an Application on clusters where ACM has placed it.
4. The hub Argo CD turns each templated Application into a ManifestWork addressed at the matching spoke (Red Hat pull model, per ADR 0018).
5. The spoke Argo CD pulls its ManifestWork through the klusterlet’s outbound gRPC channel and reconciles the contained Application against the spoke’s local API server.
6. Each Application is constrained by an AppProject (one per team) that whitelists source repos, destination namespaces, and resource kinds the tenant may create.
7. The Application’s destinationNamespace is apps-<division>-<team>-<env> (already provisioned by tenant onboarding).
8. kustomize build of the overlay renders the final Deployment (image pinned by digest), Service, Route, and any ExternalSecret, which Argo applies into the tenant namespace.

What this subsection covers

For the tenant side (what to put in apps/<team>/<app>/), see the app-repo overlay contract. For the platform side (where the ApplicationSet ships from), see federated GitLab and pull-model.

Where the pieces live

Operating boundary

Three boundaries are load-bearing and must be preserved when you change anything in this subsection:

1. Tenants cannot write Application or ApplicationSet CRs. The AppProject’s roles do not include those kinds. If a tenant needs a non-templated Application (a one-off CronJob, a Job), they file a type/enhancement issue against opp-full-plat; the platform team adds it to the ApplicationSet template or to a separate AppSet shipping from platform-gitops.
2. CI cannot push to stg/ or prd/ overlays. The Jenkins overlay-patch step and the Tekton overlay-patch step both refuse to write outside dev/. Promotion is a human-driven scripts/promote.sh run that copies the digest forward. See the promotion model for the rationale and promote.sh for the mechanics.
3. The hub Argo CD never reconciles into the spoke directly. Every tenant Application is wrapped in a ManifestWork and the spoke’s local Argo CD does the actual kustomize build and apply. This is what makes the pattern survive a hub outage and what keeps the hub CPU footprint flat as spokes scale.

Failure-mode preview

These reappear with full symptom → root cause → fix → prevention tables on the per-page sections, but flagged here so the operator knows the four most common ways this pipeline breaks:

• Argo silent / not syncing at all on a spoke after a manifest change. Usually the ACM gitops-addon installed a rogue routes.route.openshift.io CRD that shadows the aggregated OpenShift Route APIService and crashes the /openapi/v2 endpoint, which stalls every Argo sync silently. See the platform incident page — fix is oc delete crd routes.route.openshift.io.
• OutOfSync on a brand-new tenant app because the cluster-wide app-registry-pull Secret was never materialised into the new namespace. Symptom is ImagePullBackOff on the Deployment. Fix is to label the namespace apps.platform/tenant=true so the ClusterExternalSecret fans the Secret in; see 04 — ESO Secret and pullSecret.
• ApplicationSet creates an Application against the wrong cluster. Usually the Placement label matcher overlaps two clusters. Re-run the Placement evaluation: oc -n open-cluster-management get placementdecision <name> -o yaml and confirm status.decisions contains exactly the target.
• kustomize build succeeds in Argo but the rendered manifest has a tag, not a digest. The overlay was hand-edited to a newTag: after the digest was patched in by CI. Lint catches this pre-merge; the kustomize build … | grep -v @sha256 negative check in 03 — App-repo overlay contract catches it post-merge.

References

• opp-full-plat/connection-details/app-repo-contract.md (issue #182)
• opp-full-plat/connection-details/image-digest-overlay.md (issue #185)
• opp-full-plat/connection-details/promotion-model.md (issue #184)
• opp-full-plat/connection-details/app-registry-pullsecret.md (issue #172)
• opp-full-plat/adr/0014-developer-readiness-platform-contract.md
• opp-full-plat/adr/0015-federated-gitops-repo-architecture.md
• opp-full-plat/adr/0018-acm-openshift-gitops-pull-model-v6.md
• ACM gitops-addon Routes CRD lab memory (incident #153, 2026-05-10)