Brac POC — bank-payment app

End-to-end payment workflow on OpenShift — designed, built and shipped with the bank's existing pipeline. Single sign-on, canary release, browser-trusted TLS at the edge, mTLS in the cluster, live health view, and standard web hardening — all delivered through GitOps.

Demo asset for the BRAC engagement. A working payment workflow with customer and approver roles, single sign-on, a canary release of the customer page, browser-trusted TLS at the edge, mutual TLS between the internal components, a real-time health view of the backend services, and three of the standard web-hardening controls every audit asks for. Built end-to-end on top of the bank’s existing platform — no new vendor, no new toolchain.

At a glance

Where it runsOpenShift cluster spoke-dc-v6
Customer pagehttps://payments.apps.spoke-dc-v6.sub.comptech-lab.com
Approver pagehttps://approver.apps.spoke-dc-v6.sub.comptech-lab.com
Health viewhttps://payments.apps.spoke-dc-v6.sub.comptech-lab.com/obs/
LoginWSO2 Identity Server, tenant bank-payment
Browser-trusted TLSPublic Let’s Encrypt wildcard, auto-renewed
Internal TLSmTLS between the in-cluster proxy and the backends, certs issued by the lab CA
ObservabilityTwo parallel surfaces — OpenShift native (Tempo + user-workload Prometheus + LokiStack + Perses dashboard) and SigNoz on the lab VM (traces, metrics, logs, APM, alerts, SLO)
DeliveryOne Git repo → Jenkins → Nexus → Argo CD → OpenShift

What got built

Four pieces in one namespace:

PieceRoleWho uses it
payment-client-svcCustomer payment API. Owns the accounts and the transfer history.All users (under the hood)
payment-approver-svcAdmin API. Reviews and decides on pending transfers.Admin only
client-frontendCustomer web page. Balance, send-payment form, transaction history.zahid, shaikat
approver-frontendAdmin web page. Pending queue, approve / decline, audit log.admin

Who logs in

All three users share one demo password: Hkj38djf&&&.

UserAccountBalanceRole
zahid5005-0001-2350,000 BDTcustomer
shaikat5005-0004-5675,000 BDTcustomer
adminapprover

Login is single sign-on through WSO2 — the bank doesn’t manage usernames or passwords inside the app itself.

Technology choices

Every layer of the app uses something the bank’s operations team already runs in production. Nothing exotic, nothing that needs a new vendor relationship to support.

LayerWhat it usesWhy this choice
Backend servicesOpen Liberty (Java application server, MicroProfile 6.1)Same runtime the bank’s existing platform already runs. Long-term enterprise support from IBM / Red Hat. Built-in health and metrics endpoints — no extra agent required.
Web frontendsReact (single-page apps), built with Vite, styled with TailwindIndustry-standard frontend stack. Light, fast to load, easy to hand off to any frontend team.
Reverse proxy + canary + hardeningNGINXThe de-facto edge proxy. Used here for header-based canary routing, rate limiting, branded error pages, and the security headers. Zero learning curve for ops teams.
Identity & SSOWSO2 Identity ServerThe bank’s existing identity platform. Two service-provider entries, one tenant, regular OpenID Connect — no custom auth code in the apps.
Container platformOpenShift (Kubernetes-based, by Red Hat)The bank’s existing container platform. Standard images, standard deployment objects.
Public TLS certificateLet’s Encrypt via cert-managerGlobally-trusted certificate, refreshed automatically — no procurement cycle, no manual rotation.
Internal TLSLab Certificate Authority via cert-managerSigns the in-cluster mTLS certs. The CA root is already trusted by every Java pod and every node.
Source controlGitLabThe bank’s existing development pipeline.
Build pipelineJenkinsThe bank’s existing CI/CD.
Image registryNexus RepositoryThe bank’s existing artifact store. Same registry the rest of the platform pulls from.
Deployment pipelineArgo CD (GitOps)One commit to the Git repository becomes one change on the cluster, automatically. No engineer logs in to the running environment to make a change.
TracingRed Hat build of OpenTelemetry + TempoThe OpenTelemetry collector receives spans over OTLP and stores them in Tempo. Every distributed payment flow is one trace.
MetricsOpenShift user-workload PrometheusScrapes Liberty’s /metrics directly via a ServiceMonitor.
LogsOpenShift Logging + LokiStackVector collects pod stdout; Loki stores and indexes it.
DashboardsCluster Observability Operator + PersesDashboards declared as YAML, rendered inside the console at Observe → Dashboards.
APM / parallel observabilitySigNoz (on a lab VM)Receives the same spans / metrics / logs via OTLP from an in-cluster fan-out collector. Adds APM (service map, RED metrics, exceptions), trace ↔ log correlation, alerts, SLO tracking. Used side-by-side with the OpenShift native surface — same data, richer querying.

How the app was designed

A small set of decisions, made up front, that the rest of the implementation followed:

DecisionWhat it means in practice
Two backend services, one customer, one approverThe customer service is the system of record for accounts and transfers. The approver service is a thin admin facade — it reviews and decides, but doesn’t own data. Separation makes auditing simpler: every approver decision is recorded on the customer service with the admin’s identity.
Two separate web frontends, one per roleThe customer and admin experiences are different products with different audiences, different navigation, different theming. Splitting them eliminates accidental cross-exposure of admin features to customers.
Reserve-on-create, complete-on-approveWhen a customer submits a payment, the funds leave their visible balance immediately. The transfer can’t be double-spent during review. On Approve the funds land in the recipient; on Decline they’re refunded.
No service can speak for the userEvery API call carries the user’s own bearer token. When the approver service calls the customer service to settle a transfer, it forwards the admin’s token, not a service account. The audit trail captures the actual deciding human.
No new vendor, no new toolchainEvery layer (runtime, build, identity, registry, ingress) maps to something the bank already runs.

How the app is built

A single push to GitLab triggers everything else.

StepWhat happensWhere
1. Developer pushes to mainGitLab webhook firesgitlab.apps.sub.comptech-lab.com/divisions/payment/bank-payment
2. Jenkins runs the pipelineDefined in Jenkinsfile at the root of the repoJenkins (lab VM)
3. Four container images built in parallelOne per app — backend services and frontendsJenkins build agent (ct-shared-build)
4. Each image scanned for high / critical vulnerabilitiesTrivy. Build fails if any HIGH or CRITICAL CVE is found.Same agent
5. Images tagged twice and pushed:latest and :<short-commit-sha>Nexus (app-registry.apps.sub.comptech-lab.com)
6. GitOps repo updated with the new digestA small script writes the immutable digest into the Argo overlay and commits backSame pipeline
7. Argo CD reconciles the new digest onto the clusterPicks up the commit, rolls the deploymentOpenShift

The whole sequence is four to six minutes end-to-end. The build fails closed — a single critical CVE blocks the push to Nexus, which blocks the deployment. No image ever lands on the cluster without having been scanned.

GitOps deployment

The cluster’s state is a rendered copy of two Git repositories. No engineer ever runs oc apply against the running environment.

RepoWhat lives thereWho can write to it
divisions/payment/bank-paymentThe four app sources, the Containerfiles, the kustomize manifests (Deployments, Services, Routes, NGINX config, mTLS Certificates), the JenkinsfileThe payment-division developers
comptech-platform/openshift-ops/openshift-platform-gitopsTenant scaffolding (namespace, quotas, RBAC, network policy, Argo Application)Platform admins

On the cluster, Argo CD watches both repositories and continuously reconciles their contents onto OpenShift. Two important properties fall out of this:

  • Self-healing. If anything drifts from what’s in Git — someone patches a Service directly, a resource gets deleted by mistake — Argo reverts it within seconds.
  • Auditable. Every change to the running app is a Git commit, with an author, a timestamp, and a diff. Compliance gets the audit trail for free.

The deployment itself uses two-side reconciliation: the cluster declares it wants the image at digest sha256:abc…, the image already exists in Nexus, OpenShift pulls it, and rolls the deployment. The whole sequence after a Jenkins build is typically under a minute.

Traffic flow — browser to pod

A request from zahid’s browser passes through six named layers before it reaches an application pod:

HopWhat it doesWhere it lives
1. Cloudflare DNSAuthoritative DNS for comptech-lab.com. Returns NS records that delegate the sub.comptech-lab.com subdomain to the lab’s own DNS server.Public, managed by Cloudflare
2. PowerDNSAuthoritative DNS for the sub.comptech-lab.com zone. Returns the public IP of the lab’s edge load balancer for any *.apps.spoke-dc-v6.sub.comptech-lab.com query.Lab VM (pdns)
3. HAProxyEdge TCP / SNI load balancer for the whole lab. Reads the SNI hostname from the inbound TLS ClientHello and forwards the connection to the right backend — the spoke OpenShift cluster’s ingress in this case. TLS is passed through end-to-end.Lab VM (haproxy at 30.30.30.1)
4. OpenShift RouterThe cluster’s ingress. Terminates the public Let’s Encrypt TLS, looks up the matching Route, forwards the request as plain HTTP to the in-cluster Service.Cluster (openshift-ingress namespace)
5. NGINX reverse proxyThe in-cluster gateway for the customer page. Applies header-based canary routing, per-IP rate limiting, branded error pages, and the security response headers. For the live health view, also presents a client certificate over mutual TLS to the backend.Cluster (bank-payment namespace)
6. App podThe actual web frontend or backend service — Open Liberty for the APIs, httpd for the static SPA bundles.Cluster (bank-payment namespace)

The public TLS terminates at step 4. From step 5 onward, the in-cluster hop between the proxy and the backend service is also encrypted, using mTLS — see Internal mTLS below.

System view — interactive

Hint: drag any box to rearrange. Click a box to flip its colors (useful for highlighting a path you’re walking through). The lines are live — the animation shows the direction traffic flows.

Reading the diagram:

Line colourWhat it means
Solid blackUser traffic (HTTPS) — DNS → edge → in-cluster
Solid amberThe canary hop (header-triggered) and the mTLS hop (/obs/* from NGINX to Liberty)
Solid greenCross-service call (approver-svc → client-svc with the admin’s bearer forwarded)
Dashed greyDNS resolution
Dashed magentaIdentity-related — sign-in redirect, JWKS verify, cert distribution from cert-manager
Dashed amberBuild pipeline (GitLab → Jenkins → Trivy → Nexus → digest pin)
Dashed greenGitOps reconcile (Argo watching the repos, pulling images, rolling Deployments)

How a payment moves

  1. Zahid logs in on the customer page. Browser is redirected to WSO2, signs in, comes back signed in.
  2. He picks a recipient from the dropdown — accounts are listed by name and account number; balances are not exposed across customers.
  3. He submits a payment for an amount. Funds are reserved against his balance immediately.
  4. Admin logs in on the approver page. The pending queue auto-refreshes every five seconds.
  5. Admin clicks Approve. Funds move into the recipient’s balance, the transfer is logged with the admin’s name on the audit trail.
  6. Decline would refund the sender. Either decision is one-shot — the app won’t let the same transfer be settled twice.

Canary release of the customer page

Two versions of the customer page run side-by-side. Visitors get the stable v1 unless they explicitly ask for v2.

v1 (stable)v2 (canary)
AppearanceWhite backgroundLight-yellow background, “v2 (canary)” tag next to the title
Who reaches itEveryone by defaultOnly requests that carry an explicit X-Canary: v2 opt-in header
Replicas21
Switching isInstant on the next page load, no redeployInstant on the next page load

To show v2 in a demo: install the ModHeader browser extension, add a single rule (Name: X-Canary, Value: v2), reload. Disable the rule to flip back to v1.

The routing decision is visible in the x-served-by response header — open Chrome’s dev tools → Network → click the page request → Headers → Response Headers — to see which version served you.

Internal mTLS

Public traffic to the app uses a browser-trusted Let’s Encrypt certificate; that part is unchanged. The hop inside the cluster between the NGINX reverse proxy and the health-view endpoints on the backend services now uses mutual TLS — both sides authenticate with certificates, both sides verify each other’s certificate against the lab certificate authority.

Client (NGINX proxy)Server (Liberty backend)
IdentityCert issued by the lab CA, CN = client-frontend-canary-proxy.bank-payment.svc.cluster.localCert issued by the lab CA, CN = payment-client-svc.bank-payment.svc.cluster.local
Cert lifetime90 days, rotated automatically 30 days before expiry by cert-managerSame
What happens without a certThe Liberty backend refuses the connection at the TLS handshake

To prove mTLS is doing its job, two commands from a terminal:

# 1. Without the client cert — handshake rejected
oc -n bank-payment exec deploy/client-frontend-canary-proxy -- \
  curl -k --max-time 5 https://payment-client-svc:9445/health/live
# → SSL_ERROR_ZERO_RETURN

# 2. With the client cert — accepted, returns 200 + JSON
oc -n bank-payment exec deploy/client-frontend-canary-proxy -- \
  curl --cert /etc/nginx-mtls/tls.crt --key /etc/nginx-mtls/tls.key \
       --cacert /etc/nginx-mtls/ca.crt \
       https://payment-client-svc.bank-payment.svc.cluster.local:9445/health/live
# → {"status":"UP",...}

The plain-HTTP port on the backend (9080) stays open for the OpenShift platform’s own probes and for the public-facing API Routes, which are already protected by the bearer token from WSO2.

Egress controls — what the pods can talk to

By default the pods in the bank-payment namespace cannot reach anything outside their own namespace, not even the wider cluster. Every outbound destination is opened by an explicit allow-rule. This limits the blast radius of a compromised pod — a leaked credential can’t be exfiltrated to a public server, the pod can’t pivot into other tenant namespaces, and the only external destination it can reach is the bank’s own identity service.

DestinationWhy it’s neededHow it’s allowed
In-cluster DNS (openshift-dns)Resolving Service names like payment-client-svc.bank-payment.svc.cluster.localNetworkPolicy allow-egress-dns — UDP/TCP 53
Cluster monitoring (openshift-monitoring)Prometheus scrape from the platform’s monitoring stackNetworkPolicy allow-egress-cluster-monitoring — TCP 9091-9093
WSO2 Identity Server (lab VM, 160.30.63.134/138:443)Liberty fetches the JWKS signing keys here to validate every incoming user tokenNetworkPolicy allow-egress-wso2-is — restricted to those exact two IPs on TCP 443
Other pods in the same namespaceThe approver service calling the customer service, the NGINX proxy calling the backends, etc.NetworkPolicy allow-intra-namespace
The public internetNot allowed. No NetworkPolicy opens this.
Other tenant namespacesNot allowed. Pods in bank-payment cannot speak to pods in any other tenant.

The full set of NetworkPolicies for this tenant lives in the platform GitOps repository — adding a new outbound destination is a small Git commit, reviewed and audited like everything else.

Observability — two parallel surfaces

Every signal the app emits (traces, metrics, logs) is shipped to two independent observability backends in parallel — the OpenShift-native stack (in the cluster console, owned by the platform team) and SigNoz running on a lab VM (a full observability platform with APM, alerts, SLO tracking).

The split is intentional. The platform team uses OpenShift’s native view because it lives in the same console they already operate the cluster from. App developers and the on-call rotation use SigNoz because it has richer per-endpoint analytics and a side-by-side trace / log / metric view. Both surfaces show the same data — the app pushes once, an in-cluster collector fans out.

Where each signal lands

SignalOpenShift nativeSigNoz (lab VM)
TracesOpenTelemetry collector → Tempo in the tracing namespace. View at console Observe → Traces.Same span stream is also shipped over OTLP to SigNoz, which stores it in ClickHouse for richer querying.
MetricsUser-workload Prometheus scrapes Liberty’s /metrics every 30s. View at console Observe → Metrics.Liberty also pushes the same metric set over OTLP; SigNoz stores it in ClickHouse and exposes it for dashboards / alerts.
LogsOpenShift Logging’s Vector ships pod stdout to LokiStack. View at console Observe → Logs.Liberty’s MicroProfile Telemetry log handler pushes every message via OTLP with the trace_id / span_id stamped on each line, so SigNoz can jump straight from a trace to the matching log lines.

Tail sampling — kept the interesting traces, dropped the noise

The in-cluster fan-out collector applies tail-based sampling to traces before forwarding to SigNoz. Sampling decisions are made on the complete trace:

Trace shapeKept?
Contains any errored spanAlways kept
Total duration > 500 msAlways kept
All-healthy under 500 ms10% kept (probabilistic)

Net effect: anomalies survive at full fidelity, normal traffic is ten-times-cheaper to store, and the trace storage bill scales with meaningful traffic instead of raw volume.

Business-aware traces

Spans on the transfer and approve / decline endpoints carry the business context as searchable attributes — not just generic HTTP metadata:

AttributeExample
transfer.id1a2b-3c4d-…
transfer.from · transfer.tozahid · shaikat
transfer.amount · transfer.status2500.00 · PENDING
decision (on approver path)approve / decline
bank.user · bank.user.roleadmin · admin

In SigNoz, you can paste a transfer ID into the trace search and find that exact customer payment — including the cross-service hop to the approver, the ledger write, every Liberty handler — in one waterfall.

What SigNoz gives you on top of the data

The OTLP plumbing is the same as for Tempo, but SigNoz adds an APM layer that needs no extra wiring:

SigNoz featureWhat it gives the demo
Service MapAuto-drawn graph of every service-to-service call, arrow thickness = traffic volume. Visually proves the approver → client cross-call is happening.
APM / RED metrics per endpointAuto-derived from spans — request Rate, Error rate, Duration (p50/p95/p99) for every API path. No code change.
Slowest endpoints leaderboardSorted list of the slowest operations across both backends.
Exception viewWhen Liberty throws, the exception + Java stack lands directly on the span — visible in one click.
Trace → log jumpClick any span → SigNoz pulls the matching log lines automatically via trace_id. Single pane of glass.
Alerts (UI-driven)Pre-staged: p99 on /v1/transfers > 500 ms · backend down (up < 1) · heap utilization > 80 %.
SLO panels (UI-driven)“99.5% of /v1/transfers < 500 ms over 30 days” — burn-rate tracker baked in.
Saved viewsA pre-built “find a transfer by ID” view that filters across both services.

Curated dashboard (Perses, in the OpenShift console)

A pre-built Perses dashboard ships alongside the app and renders under Observe → Dashboards → Bank Payment Overview in the console. Three collapsible grids:

GridPanels
At a glanceRunning pods · Heap utilization (avg %) · JVM threads (avg) · HTTP request rate (req/s)
JVM resourcesHeap used (MB) per pod · Thread count per pod · Process CPU load (%) per pod · GC time rate
HTTP trafficRequest rate by route × status code · p95 latency by route

The dashboard YAML lives in the same repo as the app; the Cluster Observability Operator’s Perses operator reconciles it into the console.

Quick-glance in-app status page

Independent of the platform tooling and SigNoz, the app also serves a tiny status page at /obs/ directly on the customer hostname. Two cards, one per backend, refreshed every five seconds. Convenient when you want a one-click health check without leaving the app.

Metric on /obs/What it tells you
LivenessIs the service process responding at all? Red if it isn’t.
ReadinessIs the service ready to take traffic? Red during a restart.
JVM thread countSnapshot of in-flight work.
Heap used (MB)Memory pressure.
CPU (recent %)Live CPU usage.

This in-app page goes through the mTLS hop described above; the console-side observability and SigNoz go through their own collectors which run in their own namespaces / on the lab VM.

Web hardening at the edge

Three controls applied in front of every request, configured in one NGINX file:

ControlWhat it doesHow to verify in 10 seconds
HSTS (Strict-Transport-Security)Browser remembers for 1 year that this site is HTTPS-only. Even typing http:// won’t downgrade.Dev tools → Network → reload → click request → Headers → look for strict-transport-security.
Branded error pages404 / 429 / 5xx all return a bank-branded page instead of nginx’s grey default.Fire 30 rapid requests at the health view — the 429 page renders.
Per-source rate limitingCaps how fast one source can hit a path. Stops dictionary attacks on login forms and stops scrapers hammering the health view.for i in {1..30}; do curl -sS -o /dev/null -w '%{http_code} ' …/obs/client/metrics; done — you’ll see a mix of 200s and 429s.

Two additional headers travel alongside HSTS: X-Content-Type-Options: nosniff (browsers can’t misinterpret file types) and Referrer-Policy: strict-origin-when-cross-origin (other sites don’t see internal paths in their referrer logs).

Where everything lives

URL
Customer pagehttps://payments.apps.spoke-dc-v6.sub.comptech-lab.com
Approver pagehttps://approver.apps.spoke-dc-v6.sub.comptech-lab.com
In-app health viewhttps://payments.apps.spoke-dc-v6.sub.comptech-lab.com/obs/
Console dashboardOpenShift console → Observe → Dashboards → Bank Payment Overview
SigNoz UIhttps://signoz.apps.sub.comptech-lab.com — Services / Traces / Logs / Metrics / Alerts / Dashboards
Customer APIhttps://client-api.apps.spoke-dc-v6.sub.comptech-lab.com
Approver APIhttps://approver-api.apps.spoke-dc-v6.sub.comptech-lab.com
WSO2 IS consolehttps://is.apps.sub.comptech-lab.com/t/bank-payment/console
Git repo (app)gitlab.apps.sub.comptech-lab.com/divisions/payment/bank-payment
Git repo (platform / Argo)gitlab.apps.sub.comptech-lab.com/comptech-platform/openshift-ops/openshift-platform-gitops
Image registryapp-registry.apps.sub.comptech-lab.com/bank-payment/*

Roadmap (intentionally not yet built)

  • Persistent storage for accounts + transfer history. The demo uses in-memory state; a service restart wipes the ledger. JDBC + PostgreSQL or a PVC-backed JSON snapshot is the obvious next step.
  • API Gateway in front of the two services — the gateway definitions are already in the repo, waiting on WSO2 APIM.
  • Service mesh for fleet-wide mTLS and a richer canary story beyond a single header rule.
  • Audit event stream — every approver decision emitted to Kafka for downstream reconciliation, plus the same event flowing through the OpenTelemetry pipeline for correlation with the trace data.

Last reviewed: 2026-05-12