Sample: Mesh + Tempo trace

Two-service app (front → backend → CNPG) labelled for ambient service mesh, OTel-instrumented, with traces visible in Tempo and Kiali topology rendering the edge.

Deploy in 5 commands

Prerequisites: tenant onboarding complete for apps-platform-mesh-trace-dev; the platform’s Istio ambient mesh + the Tempo operator + Kiali are installed; the tenant namespace is labelled istio.io/dataplane-mode=ambient for ambient mode participation; CNPG operator is installed (the backend writes to a Postgres Cluster as in sample 03).

# 1. Clone the platform app monorepo and confirm the sample is present.
git clone http://gitlab.sub.comptech-lab.com/divisions/platform/platform-apps-monorepo.git
ls platform-apps-monorepo/apps/team-platform/mesh-trace-sample

# 2. Wait for Argo to roll out both Deployments + the CNPG Cluster (~ 3 min).
oc -n apps-platform-mesh-trace-dev rollout status deploy/front --timeout=180s
oc -n apps-platform-mesh-trace-dev rollout status deploy/backend --timeout=180s
oc -n apps-platform-mesh-trace-dev wait cluster/mt-pg --for=condition=Ready --timeout=300s

# 3. Generate trace traffic.
ROUTE=$(oc -n apps-platform-mesh-trace-dev get route front -o jsonpath='{.spec.host}')
for i in $(seq 1 20); do curl -fsS -X POST "https://${ROUTE}/items" \
  -H 'Content-Type: application/json' -d "{\"name\":\"item-$i\"}"; done

# 4. Open Kiali topology — expect an edge `front → backend → mt-pg`.
oc -n istio-system get route kiali -o jsonpath='https://{.spec.host}/console/graph/namespaces/?namespaces=apps-platform-mesh-trace-dev&duration=300{"\n"}'

# 5. Open Tempo — search by service name `front` and confirm a trace with three spans.
oc -n openshift-tempo-operator get route tempo -o jsonpath='https://{.spec.host}/{"\n"}'

Expected: in Kiali, a graph showing unknown → front → backend → mt-pg with throughput numbers. In Tempo, a trace with three spans (front: POST /items → backend: POST /items → backend: INSERT items).

What this sample demonstrates

The whole observability triangle — service mesh + traces + topology — in one tenant deployment.

Per issue #204 (DEV-OCP-5.5).

It exercises:

• Ambient mode service mesh (Istio ambient, no sidecar injection). The namespace label istio.io/dataplane-mode=ambient opts in.
• OTel instrumentation — the front and backend containers ship with the @opentelemetry/sdk-node SDK (Node) configured to send OTLP/gRPC to http://otel-collector.istio-system:4317.
• The OTel Collector running as a DaemonSet (part of the platform’s observability stack), forwarding OTLP to the TempoStack.
• TempoStack backed by a NooBaa OBC (S3-compatible storage on ODF). The ESO bridge translates OBC-emitted Secret keys into the lowercase keys TempoStack expects — see platform memory project_obc_to_operand_secret_bridge.md.
• Kiali topology — reads from Prometheus (service mesh metrics) and Tempo (trace edges) to render the per-namespace graph.
• CNPG Cluster mt-pg — backend writes go to Postgres; trace spans the DB call.

Whiteboard — request and trace flow

Solid black lines = request path. Dashed purple lines = OTel span emission. The front and backend Pods emit spans to the OTel Collector, which forwards via OTLP to TempoStack. Kiali reads Tempo + Prometheus to render the topology.

Repo layout

apps/team-platform/mesh-trace-sample/
  base/
    kustomization.yaml
    deployment-front.yaml
    deployment-backend.yaml
    service-front.yaml
    service-backend.yaml
    route.yaml                           # only on `front`
    cluster.yaml                         # CNPG Cluster mt-pg
    backup-schedule.yaml
    configmap-front.yaml                 # has BACKEND_URL=http://backend.<ns>.svc:8080
    configmap-backend.yaml               # has PGHOST=mt-pg-rw, PGDATABASE=mesh_trace
    externalsecret-minio-backup.yaml
  overlays/
    dev/
      kustomization.yaml                 # adds istio.io/dataplane-mode=ambient label via patchTransformer
    stg/
      kustomization.yaml
    prd/
      kustomization.yaml

Manifest highlights

Ambient mesh participation

In the overlay (per-env so dev/stg can have different mesh policies than prd):

# apps/team-platform/mesh-trace-sample/overlays/dev/kustomization.yaml
namespace: apps-platform-mesh-trace-dev

commonLabels:
  apps.platform/env: dev
  apps.platform/division: platform
  app.kubernetes.io/instance: mesh-trace-sample-dev
  app.kubernetes.io/version: "REPLACED_BY_CI"

  # Ambient mesh participation — the only addition vs. a non-mesh app.
  istio.io/dataplane-mode: ambient

resources:
  - ../../base

images:
  - name: app-registry.apps.sub.comptech-lab.com/team-platform/mesh-trace-sample-front
    newName: app-registry.apps.sub.comptech-lab.com/team-platform/mesh-trace-sample-front
    digest: sha256:REPLACED_BY_CI_FRONT
  - name: app-registry.apps.sub.comptech-lab.com/team-platform/mesh-trace-sample-backend
    newName: app-registry.apps.sub.comptech-lab.com/team-platform/mesh-trace-sample-backend
    digest: sha256:REPLACED_BY_CI_BACKEND

The label istio.io/dataplane-mode=ambient on the namespace is what opts every Pod in the namespace into the ambient mesh. No sidecar injection; the ztunnel DaemonSet on each node handles L4 mTLS for Pods in this namespace.

The label is set via a Namespace patch in the tenant template (not in the overlay; namespaces are platform-owned). Tenants who want ambient mesh ask in the onboarding MR; the platform admin adds the label.

OTel instrumentation (front, Node.js excerpt)

// server.js (front)
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const sdk = new NodeSDK({
  serviceName: 'front',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector.istio-system.svc:4317'
  }),
  instrumentations: [
    require('@opentelemetry/instrumentation-http').HttpInstrumentation(),
    require('@opentelemetry/instrumentation-express').ExpressInstrumentation(),
  ],
});
sdk.start();

const app = require('express')();
app.post('/items', async (req, res) => {
  const r = await fetch(process.env.BACKEND_URL + '/items', { method: 'POST', body: JSON.stringify(req.body), headers: { 'content-type': 'application/json' } });
  res.status(r.status).send(await r.text());
});
app.listen(8080);

The SDK auto-injects W3C trace context headers (traceparent, tracestate) on outbound HTTP, so when front calls backend, the spans share a trace_id.

Backend Postgres span instrumentation

The backend uses @opentelemetry/instrumentation-pg to add a span around every Postgres query, so the trace shows the DB call as a child span of the backend: POST /items span.

// server.js (backend)
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const sdk = new NodeSDK({
  serviceName: 'backend',
  traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector.istio-system.svc:4317' }),
  instrumentations: [
    require('@opentelemetry/instrumentation-http').HttpInstrumentation(),
    require('@opentelemetry/instrumentation-pg').PgInstrumentation(),
  ],
});
sdk.start();

Verifying mesh + traces

Ambient mesh: confirm ztunnel handles the traffic

NS=apps-platform-mesh-trace-dev

# 1. Confirm the namespace is in ambient mode.
oc get ns $NS -o jsonpath='{.metadata.labels.istio\.io/dataplane-mode}{"\n"}'
# Expected: ambient

# 2. Pods should NOT have a sidecar container.
oc -n $NS get pod -l app.kubernetes.io/name=front -o jsonpath='{.items[0].spec.containers[*].name}{"\n"}'
# Expected: app   (NOT app istio-proxy)

# 3. ztunnel sees the Pods.
oc -n istio-system get pod -l app=ztunnel -o wide

# 4. Verify L4 mTLS by checking ztunnel's logs after sending a request.
oc -n istio-system logs -l app=ztunnel --tail=50 | grep mesh-trace

Traces: open Tempo and find the trace

# Generate traffic.
for i in $(seq 1 5); do curl -fsS -X POST "https://${ROUTE}/items" -H 'Content-Type: application/json' -d "{\"name\":\"trace-$i\"}"; done

# Open Tempo UI via the OCP Console (Observe → Traces) or directly:
oc -n openshift-tempo-operator get route tempo \
  -o jsonpath='https://{.spec.host}/search?spanName=POST+%2Fitems{"\n"}'

In Tempo:

• Service front shows POST /items spans.
• Each span has a child backend: POST /items (linked via traceparent headers).
• Each backend span has a child pg.query span showing the Postgres INSERT INTO items.

Kiali topology

# Kiali Route.
oc -n istio-system get route kiali \
  -o jsonpath='https://{.spec.host}/console/graph/namespaces/?namespaces='$NS'&duration=300{"\n"}'

In Kiali:

• Open the namespace graph for apps-platform-mesh-trace-dev.
• See the edge from front → backend with request throughput.
• The mt-pg-rw Service is shown as a sink edge from backend.

Smoke validation

NS=apps-platform-mesh-trace-dev

# 1. Argo Application Synced/Healthy.
oc -n openshift-gitops get app team-platform-mesh-trace-sample-dev \
  -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{"\n"}'

# 2. Both Deployments rolled out.
oc -n $NS get deploy

# 3. CNPG Cluster Ready.
oc -n $NS get cluster mt-pg

# 4. Pods do NOT have sidecars (ambient mode).
oc -n $NS get pod -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.containers[*].name}{"\n"}{end}'

# 5. Send a request and confirm the row exists.
ROUTE=$(oc -n $NS get route front -o jsonpath='{.spec.host}')
curl -fsS -X POST "https://${ROUTE}/items" -H 'Content-Type: application/json' -d '{"name":"smoke"}'
curl -fsS "https://${ROUTE}/items"

Failure modes

Adapting for your own app

References

• Issue #204 (DEV-OCP-5.5) — sample closure.
• Istio ambient mesh docs — istio.io/dataplane-mode=ambient.
• TempoStack operator docs — tempo.grafana.com/v1alpha1/TempoStack.
• Kiali docs — namespace graph + TraceQL integration.
• OpenTelemetry Node SDK docs — @opentelemetry/sdk-node, @opentelemetry/instrumentation-pg.
• Platform memory project_obc_to_operand_secret_bridge.md — ESO key-name bridge for Tempo storage.
• ADR 0014 — developer-readiness-platform-contract.