CompTech AIOps - backup and restore baseline

Public-safe summary of the ze-tower control-plane backup and restore baseline.

The ze-tower control plane needs a recovery baseline before production-changing agent authority or serious RAG ingestion is enabled.

The canonical decision is ADR 0017 in the comptech-aiops repository. This page is the public-safe summary.

Recovery Model

Backup Tooling

The Phase 1 direction uses Restic for encrypted backup repositories stored in the AIOps MinIO backup bucket.

This matters because backup payloads can contain sensitive runtime state. They must be encrypted before upload.

The public docs can name the tool and bucket role, but they do not publish:

• passwords;
• access keys;
• private backup repository credentials;
• sensitive snapshot IDs;
• raw backup manifests;
• operational evidence that exposes private details.

Stateful Services

The backup baseline covers:

• platform Postgres;
• Temporal Postgres;
• NetBox Postgres;
• Qdrant;
• n8n data;
• Grafana data;
• Prometheus data;
• Loki data;
• Tempo data;
• NetBox Redis.

The priority is not equal for all state. Databases and future source-of-truth services have higher restore priority than metrics, logs, traces, and rebuildable vector indexes.

Restore Drill Gate

Production-changing agent capability required at least one completed restore drill before moving forward.

The restore drill must use a separate target path, not the live deployment root. The drill should prove that:

• repository source can be restored or synced;
• runtime configuration can be regenerated without printing secrets;
• backup snapshots can be selected;
• databases and volumes can be restored;
• Compose validation works;
• health checks and data checks can be captured safely.

Implementation Status

The first encrypted backup implementation is complete in the canonical comptech-aiops repository under issue #22.

Public-safe outcomes:

• Restic was installed and validated on ze-tower.
• Backup runtime settings were rendered from GitHub Secrets into a non-repository runtime file.
• The encrypted Restic repository was initialized in the AIOps backup bucket.
• The backup workflow created logical dumps for Postgres-backed services.
• The backup workflow archived selected non-Postgres Docker volumes.
• The first restore drill completed against a non-production target.
• The drill verified checksums, compressed database dumps, and volume archives.

Private values, endpoint details, raw evidence manifests, and secret material are intentionally excluded from this public page.

Retention Policy

The canonical retention decision is comptech-aiops ADR 0018.

Phase 1 keeps:

Retention is scoped to ze-tower control-plane snapshots. The workflow defaults to dry-run mode, which records what would be kept or forgotten without deleting backup data.

Apply mode is explicit. It runs the policy and prunes unreferenced encrypted data only after the dry-run evidence has been reviewed.

The first retention dry-run and apply-mode run completed successfully. The current public-safe outcome is that the proven restore snapshot remains retained.

MinIO Protection Boundary

The canonical MinIO protection decision is comptech-aiops ADR 0019.

The encrypted ze-tower backups live in the AIOps MinIO backup bucket, so the backup workflow is not a complete disaster recovery plan for MinIO itself. Critical AIOps object buckets need replication or export to a separate target outside the primary MinIO server and outside the same physical failure domain.

Initial public-safe priority:

Acceptable future implementation paths are bucket replication to another S3-compatible target, scheduled object mirroring to independent storage, or offline encrypted object export when online replication is unavailable.

The implementation target is still pending. The next step is to choose the independent target, configure replication or export, and run a restore drill from that copy.

Open Follow-Ups

• Choose and implement the independent MinIO replication or export target from ADR 0019.