NooBaa OBC -> operand storage-Secret bridge

ObjectBucketClaim outputs Secret keys (AWS_ACCESS_KEY_ID...) that don't match the LokiStack/TempoStack storage-Secret schema (access_key_id...); ESO with templating is the bridge.

ODF NooBaa MCG is the in-cluster S3 backend for Loki, Tempo, and Quay on spoke-dc-v6. NooBaa’s ObjectBucketClaim produces a Secret with AWS_*-shaped keys. The downstream operands expect lowercase keys (endpoint, bucketnames, access_key_id, access_key_secret). The shapes don’t match, and the operand silently sits Warning Degraded until a bridge is added.

The bridge pattern is an ExternalSecret with the Kubernetes provider templating the operand-shape Secret from the OBC outputs.

Symptom

A freshly-installed LokiStack or TempoStack with the OBC’s Secret named as spec.storage.secret.name reports Warning Degraded:

K=/home/ze/.kube/configs/spoke-dc-v6.kubeconfig

# LokiStack:
oc --kubeconfig "$K" -n openshift-logging get lokistack -o yaml \
  | grep -A5 conditions
# - lastTransitionTime: "..."
#   message: 'storage secret "obc-loki" has missing keys: [endpoint bucketnames access_key_id access_key_secret]'
#   reason: ConditionReason
#   status: "True"
#   type: Warning

# TempoStack:
oc --kubeconfig "$K" -n openshift-tracing get tempostack -o yaml \
  | grep -A5 conditions
# - lastTransitionTime: "..."
#   message: '...invalid storage secret keys...'
#   status: "True"
#   type: Warning

The OBC itself is healthy:

oc --kubeconfig "$K" -n <operand-ns> get objectbucketclaim
# NAME           PHASE     STATUS   AGE
# tempo-traces   Bound              ...

oc --kubeconfig "$K" -n <operand-ns> get secret <obc-name> \
  -o jsonpath='{.data}' | base64 -d 2>/dev/null \
  || oc --kubeconfig "$K" -n <operand-ns> get secret <obc-name> -o yaml \
       | grep -A4 data:
# data:
#   AWS_ACCESS_KEY_ID: <b64>
#   AWS_SECRET_ACCESS_KEY: <b64>

NooBaa is producing keys; the operand is asking for different keys.

Root cause

When you create an ObjectBucketClaim, NooBaa creates two companion resources alongside the bucket:

Resource	Keys
`Secret` named after the OBC	`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
`ConfigMap` named after the OBC	`BUCKET_HOST`, `BUCKET_NAME`, `BUCKET_PORT`, `BUCKET_REGION`, `BUCKET_SUBREGION`

The bucket coordinates (host, name, port) end up in the ConfigMap, not the Secret.

The downstream operands expect a different shape — a single Secret with all required fields:

LokiStack.spec.storage.secret (type s3):

endpoint: "https://s3.openshift-storage.svc"
bucketnames: "<bucket-name>"
access_key_id: "<key-id>"
access_key_secret: "<secret>"

TempoStack.spec.storage.secret (type s3) has the same expectation.

Quay uses its own config-bundle format (quay-config-bundle-secret with a config.yaml field). The bridge pattern below covers the LokiStack / TempoStack case; Quay needs a different shape (populated from Vault secret/ocp/spoke-dc-v6/quay/config-bundle).

The root cause is upstream operator design disagreement:

NooBaa expects consumers to read both the Secret and the ConfigMap (the AWS-CLI tradition).
LokiStack and TempoStack expect a single self-contained Secret in the lowercase-key shape (the cloud-vendor template tradition).
Neither is going to change in the near term.

Fix

The bridge is an ExternalSecret using the ESO kubernetes provider (not the Vault provider). It reads from the OBC’s Secret, templates a new Secret in the operand shape, and writes it back into the same operand namespace.

The file at clusters/spoke-dc-v6/platform-services/tracing/externalsecret-tempo-storage.yaml is the working pattern (MR !43). Adapt for Loki under clusters/spoke-dc-v6/platform-services/logging/externalsecret-loki-storage.yaml (the open backport tracked under #233).

Four resources land together:

# ServiceAccount the ESO kubernetes provider uses to read the OBC's Secret.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-obc-reader
  namespace: openshift-tracing
---
# Role granting read on the OBC's Secret (and the ConfigMap if needed).
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eso-obc-reader
  namespace: openshift-tracing
rules:
  - apiGroups: [""]
    resources: ["secrets", "configmaps"]
    resourceNames: ["tempo-traces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eso-obc-reader
  namespace: openshift-tracing
subjects:
  - kind: ServiceAccount
    name: eso-obc-reader
    namespace: openshift-tracing
roleRef:
  kind: Role
  name: eso-obc-reader
  apiGroup: rbac.authorization.k8s.io
---
# SecretStore using the kubernetes provider, scoped to the operand namespace.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: obc-tempo-traces
  namespace: openshift-tracing
spec:
  provider:
    kubernetes:
      remoteNamespace: openshift-tracing
      server:
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
          namespace: openshift-tracing
      auth:
        serviceAccount:
          name: eso-obc-reader
---
# ExternalSecret that templates the operand-shape Secret.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: tempo-storage
  namespace: openshift-tracing
spec:
  refreshInterval: "5m"
  secretStoreRef:
    kind: SecretStore
    name: obc-tempo-traces
  target:
    name: tempo-storage
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        endpoint: "https://s3.openshift-storage.svc"
        bucketnames: "tempo-traces"
        access_key_id: "{{ .access_key_id }}"
        access_key_secret: "{{ .access_key_secret }}"
  data:
    - secretKey: access_key_id
      remoteRef:
        key: tempo-traces
        property: AWS_ACCESS_KEY_ID
    - secretKey: access_key_secret
      remoteRef:
        key: tempo-traces
        property: AWS_SECRET_ACCESS_KEY

Then the operand CR references the templated Secret:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: lab
  namespace: openshift-tracing
spec:
  storage:
    secret:
      name: tempo-storage    # the ESO-templated secret, not the OBC's
      type: s3

Validation:

# The templated secret exists with the expected keys:
oc --kubeconfig "$K" -n openshift-tracing get secret tempo-storage -o yaml \
  | grep -E '^\s+(endpoint|bucketnames|access_key_id|access_key_secret):'

# The operand recovers:
oc --kubeconfig "$K" -n openshift-tracing get tempostack \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'
# Every entry True

# A test trace lands and is queryable (via OTel HTTP -> Tempo gateway)

For Loki, substitute namespaces and OBC names, and reference loki-storage as the templated secret name in LokiStack.spec.storage.secret.name. The backport is open under #233.

Why the endpoint is hardcoded

The Tempo bridge hardcodes endpoint: "https://s3.openshift-storage.svc" rather than reading BUCKET_HOST from the OBC’s ConfigMap. The reason is a limitation of the ESO Kubernetes provider: it can only read from Secrets, not ConfigMaps, so the BUCKET_HOST value can’t be pulled in via a data: reference.

Workarounds considered:

Load the ConfigMap via a second data: entry — not supported by the ESO Kubernetes provider.
Use a kustomize substitution — adds a build-time step but doesn’t keep dynamic the way NooBaa expects.
Hardcode the endpoint to the well-known in-cluster S3 service — chosen, because it is stable across NooBaa upgrades and the bucket lives in the same cluster.

Hardcoding the endpoint is acceptable for in-cluster use. If the lab ever points Loki/Tempo at an external S3 endpoint, revisit the design.

Why the bucket name is hardcoded

The OBC’s spec.bucketName is deterministic (e.g., bucketName: tempo-traces in the OBC YAML). NooBaa auto-prefixes bucket names with a UUID by default; setting bucketName explicitly disables that prefix. The ExternalSecret template can then hardcode the bucket name rather than reading it from the ConfigMap.

If you forget bucketName in the OBC, the actual bucket will be named tempo-traces-<uuid> and the templated secret will point at the wrong bucket. The OBC spec must set bucketName for this pattern to work.

Prevention

Three layers:

The bridge pattern is the contract for new OBC consumers. When adding a new operand that reads from a NooBaa OBC, copy the Tempo file pattern into the new operand’s platform-services/<area>/ directory. The CHANGELOG entry should reference this incident page so future operators can find the pattern.
OBC spec always sets bucketName. A deterministic bucket name is the only way the templated secret can hardcode it. Add the bucketName field to the OBC contract in connection-details/openshift-spoke-dc-v6.md ODF section.
Loki backport is the open follow-up under #233. The LokiStack landed in W2.B (!41) before the bridge pattern was extracted, and is currently Warning Degraded until !43-equivalent for Loki merges. Track in routine tasks.

A related design question: why ESO Kubernetes provider instead of a controller of our own? Two reasons. First, ESO is already in the cluster for Vault-backed secrets — adding a SecretStore is cheap. Second, the Kubernetes provider’s templating is exactly the missing piece. A custom controller would duplicate that for one use case.

References

clusters/spoke-dc-v6/platform-services/tracing/externalsecret-tempo-storage.yaml
MR: !43 on platform-gitops (the Tempo bridge landing)
Issue: #233 (Loki backport — open)
opp-full-plat/connection-details/openshift-spoke-dc-v6.md (ODF + OBC inventory)
LokiStack and TempoStack API docs (Red Hat OpenShift Logging / Tracing)