Path A — Jenkins / Trivy / Nexus

The Jenkins-on-a-VM build path: how Jenkins pulls source from GitLab, builds with podman/buildah on the agent, scans with the shared Trivy VM, pushes to Nexus app-registry, uploads evidence to MinIO, and ends by writing a digest patch MR into the app GitOps repo. Includes a side-by-side comparison against Path B.

This page is the operator and developer reference for Path A — the Jenkins-based build path. It is the first build path to land in the lab (Jenkins is already live; Tekton is staging). It is the OCP counterpart to the existing docker-runtime-vm Jenkins chain documented in connection-details/jenkins.md: same Jenkins service, same agent, same Trivy / Nexus / MinIO infrastructure, same operator credentials. The only difference is the final step: docker-runtime-vm deploys via SSH to a VM; OCP deploys via a Git commit that Argo picks up.

Tracked under:

• DEV-OCP-3A.1 (#187) — extend Jenkins build jobs to target OCP.
• DEV-OCP-3A.2 (#188) — Jenkins → GitOps overlay digest-patch step.

Path A vs Path B — side-by-side

Before diving into Path A, here is the structural comparison against Path B (which Section 03-build-paths/02 covers in detail). Both paths land on the same digest patch in the same overlay file; the difference is where the build runs and which image registry it pushes to.

The defining symmetry: Argo CD reads apps/<team>/<app>/overlays/dev/kustomization.yaml and cannot tell which path produced the commit. That is the design intent — both paths are pluggable behind the same GitOps contract.

Path A architecture

flowchart LR
  dev[Developer] -->|git push main| gitlab[(GitLab CE<br/>gitlab.apps.sub.comptech-lab.com)]
  gitlab -->|push webhook<br/>notifyCommit| jenkins[Jenkins 2.555.1<br/>jenkins-0.sub]
  jenkins -->|schedules| agent[Build Agent<br/>jenkins-agent-0<br/>label: developer-build]
  agent -->|podman pull| nexus_pull[(docker-group.*<br/>base images)]
  agent -->|trivy --server| trivy[(Trivy VM)]
  agent -->|podman push| nexus_push[(app-registry.*<br/>tenant images)]
  agent -->|mc cp| minio[(MinIO<br/>developer-ci-evidence)]
  agent -->|git push| monorepo[(app-monorepo<br/>apps/&lt;team&gt;/&lt;app&gt;/overlays/dev/)]
  monorepo -->|hub Argo reads| argo[OpenShift GitOps<br/>on hub-dc-v6]
  argo -->|pull model<br/>ACM addon| spoke[Spoke Argo<br/>on target cluster]
  spoke -->|kustomize apply| ocp[OCP Deployment<br/>apps-&lt;division&gt;-&lt;team&gt;-&lt;env&gt; ns]

The build agent is a single, long-running VM with warm caches:

• Maven local repo (~/.m2/repository) — saves 30-90 seconds per Liberty/Spring build.
• Buildah layer cache — saves UBI base layer downloads.
• /opt/ops-workspace/scripts/ checkout — provides update-overlay-digest.sh and promote.sh.

For builds that benefit from the warm cache (Liberty + Maven, Spring Boot + Maven, Node.js + npm), Path A is faster cold-to-cold than a fresh Tekton pod. For builds that elastically scale across many cluster nodes, Path B’s per-build pod model wins. The choice is captured in the decision matrix.

Tooling on jenkins-agent-0

Skopeo presence was rechecked on 2026-05-09 and is at /usr/bin/skopeo. Trivy server URL is fronted by HAProxy on the Trivy VM.

Credentials

All credentials referenced below live in the Jenkins credential store at https://jenkins.apps.sub.comptech-lab.com/credentials/. Local custody copies live under opp-full-plat/secrets/jenkins/ (Git-ignored). Do not paste values into this file, into tickets, or into chat.

gitlab-mavenbot-pat may be replaced by a dedicated apps-overlay-bot PAT scoped only to the app monorepo when the lab grows past the demo phase.

Stages

The OCP-targeting Jenkins pipeline has eight user-visible stages plus a post {} cleanup block. Each stage has a deterministic failure signature.

1. Prepare Workspace

Verifies podman, skopeo, trivy, mc, and the ops-workspace script set are present on the agent. Sets XDG_RUNTIME_DIR to a workspace-local directory so rootless Podman doesn’t fight with other concurrent users on the agent.

2. Checkout App Source

checkout scm against the tenant repo. The job’s SCM credential is the read-only deploy token for that GitLab project (not the overlay-bot PAT — that’s used only for the digest-patch push).

3. Registry Login

podman login against both docker-group.* and app-registry.* using nexus-jenkinsbot.

4. Build Image

Loads lib/build-<LANGUAGE>.groovy from the shared library and runs the language-specific build. This is the only stage that varies between Liberty / Node / Spring Boot.

5. Trivy Scan Gate

Saves the local image as a tar, runs Trivy in server mode against the Trivy VM, fails the pipeline on any CRITICAL CVE. Produces trivy-scan.json and trivy-scan.txt.

Severity policy (identical for both paths, codified in ci-evidence-schema.md):

A CRITICAL failure is not a pipeline bug. The post { unsuccessful {} } block deletes the rejected image from Nexus so a vulnerable manifest is never addressable.

6. Push to Nexus app-registry

Pushes the locally-built image and captures the digest from both the local push and a skopeo inspect against the registry. The two must match; mismatch indicates registry rewriting.

7. Publish Evidence to MinIO

Uploads build.log, sbom.spdx.json, trivy-scan.json/.txt, image-digest.txt, image-registry-digest.txt under developer-ci-evidence/<team>/<app>/<git-sha>/. Also archives the same artifacts on the Jenkins build.

The schema is codified in ci-evidence-schema.md (DEV-OCP-3.7, #195) and is identical between Path A and Path B.

8. Patch GitOps Overlay

Clones the app monorepo, invokes:

ops-workspace/scripts/update-overlay-digest.sh \
  team-<TEAM> <APP> <env> <digest>

and pushes the commit bump: team-<TEAM>/<APP> <env> @<sha256-short> to APP_MONOREPO_BRANCH (typically ci/dev/<short-sha>). The push uses gitlab-mavenbot-pat embedded into the clone URL on the fly; the token is never logged because the sed rewrite happens inside the shell step before the git call.

9. Promote (optional)

Runs only when both PROMOTE_FROM and PROMOTE_TO are set on the build. Reuses the clone from stage 8, invokes:

ops-workspace/scripts/promote.sh \
  team-<TEAM> <APP> <PROMOTE_FROM> <PROMOTE_TO>

and pushes promote: team-<TEAM>/<APP> <from> @<sha> -> <to>.

Argo sync after the push

Once the digest patch lands on main, the platform ApplicationSet (DEV-OCP-2.2 / #183) sees the new commit on its next refresh (default 3 minutes; can be forced with argocd app get team-<TEAM>-<APP>-dev --refresh). The Argo app then syncs the Deployment, which causes the kubelet to pull the new image by digest from Nexus.

Live validation

Live validation on 2026-05-09 of the existing openliberty-readiness-probe-image-build job (the docker-runtime variant that maps to the same chain):

• Build #8 completed successfully
• Ran on jenkins-agent-0
• Cloned divisions/sandbox/openliberty-readiness-probe
• Pushed app-registry.apps.sub.comptech-lab.com/smoke/readiness-probe:build-8
• Enforced Trivy gate, archived reports, uploaded release evidence to MinIO
• Nexus returned HTTP 200 for the pushed manifest

The OCP variant of this job (with stage 8 and 9 enabled) shares all stages 1-7 with the docker-runtime variant. Migration is a job-template parameter change.

When to choose Path A

Choose Path A when:

• The tenant already maintains a Jenkinsfile and a developer-build job that targets docker-runtime-vm; extending it to also write the GitOps patch is the lowest-effort change.
• The build needs tools that are easier to install once on jenkins-agent-0 than to package into an OCP container (licensed compilers, large local caches).
• There is no requirement to run the build inside the target OpenShift cluster (no Tekton-based supply-chain attestation gate yet).
• The build benefits from warm mvn / podman layer caches.
• The team operates under an air-gapped change-window and wants the build to survive an OCP outage.

The full decision matrix with example apps is in 06-path-decision-matrix.mdx.

Health-check commands

Cold-start sanity, run from any host that can resolve lab DNS and has ~/.netrc-jenkins configured:

# Jenkins agent online?
curl --netrc-file ~/.netrc-jenkins -fsS \
  https://jenkins.apps.sub.comptech-lab.com/computer/jenkins-agent-0/api/json \
  | jq '{offline: .offline, idle: .idle}'

# Latest OCP-targeted build status (replace job name)?
curl --netrc-file ~/.netrc-jenkins -fsS \
  https://jenkins.apps.sub.comptech-lab.com/job/team-platform-sample-ocp-build/lastBuild/api/json \
  | jq '{number, result, duration, building}'

# Most recent overlay patch landed in the monorepo?
git -C /tmp/app-monorepo-check fetch --quiet origin main
git -C /tmp/app-monorepo-check log --oneline -5 origin/main \
  -- apps/team-platform/sample/overlays/dev/kustomization.yaml

References

• connection-details/jenkins-ocp-path.md — operator-level reference for Path A (this page is the public-doc version)
• connection-details/jenkins.md — Jenkins service endpoint, credentials, smoke jobs
• connection-details/app-repo-contract.md — overlay shape this path writes to (#182)
• connection-details/image-digest-overlay.md — digest patch convention (#185)
• connection-details/promotion-model.md — build-once / promote-by-digest model (#184)
• connection-details/ci-evidence-schema.md — evidence schema (#195)
• connection-details/nexus.md — Nexus endpoint, roles
• templates/jenkinsfiles/README.md — Jenkinsfile templates and smoke tests
• adr/0009-jenkins-single-vm.md — why Jenkins is a single VM
• adr/0014-developer-readiness-platform-contract.md
• adr/0018-acm-openshift-gitops-pull-model-v6.md
• adr/0019-nexus-only-image-supply-chain.md
• DEV-OCP issues: #187 (extend Jenkins jobs to OCP), #188 (overlay digest-patch step)