CompTech AIOps - current AIOps architecture
Public-safe summary of the custom agentic AIOps architecture now tracked in comptech-aiops.
The current architecture is a custom agentic AIOps control plane. It is not an OpenShift-only project, and it does not use Open WebUI or Ollama as platform components.
What The Platform Is For
The platform is intended to help manage:
- company development workflows;
- datacenter platforms;
- OpenShift and Kubernetes clusters;
- hypervisors and VM estates;
- bare-metal servers;
- network devices;
- UPS, generator, and cooling systems;
- storage systems;
- internal software platforms.
Target infrastructure remains outside ze-tower. The AIOps control plane reaches targets only through approved integrations, policy-controlled tools, and documented access paths.
Runtime Components
| Layer | Current decision |
|---|---|
| Portal | Custom portal from day 1 |
| Frontend | React, Vite, TypeScript, TanStack, Tailwind; first operator portal MVP and chat console implemented |
| Backend API | Go with Gin |
| Agent runtime | LangGraph supervisor and specialist agents |
| Durable workflows | Temporal |
| Policy | OPA |
| Tool boundary | MCP gateway |
| Model gateway | LiteLLM |
| Model serving | vLLM primary, TGI-class runtime optional |
| Retrieval | Deterministic local embeddings, Qdrant vector index, Postgres metadata, MinIO object storage |
| Workflow glue | n8n |
| Worker tools | Codex CLI and Claude CLI as future policy-controlled tools |
Agent Shape
The agent model should be multi-agent, not a single unrestricted shell bot.
The expected pattern is:
- supervisor agent for planning and task routing;
- specialist agents for coding, infrastructure, RAG ingestion, documentation, observability, and operations;
- Temporal workflows for durable execution and approval waits;
- OPA decisions before tool actions;
- MCP gateway for controlled tool access;
- evidence capture for operator review.
Production-changing autonomy is not enabled in Phase 1. Tool presence does not grant production authority.
Service Contract Baseline
The first contract baseline is defined in the canonical repo under ADR 0020.
Public-safe boundaries:
| Boundary | Contract direction |
|---|---|
| Portal frontend | Calls only the portal API. |
| Portal API | Operator-facing /api/v1 boundary. |
| Agent orchestrator | Internal planning and durable task coordination. |
| MCP gateway | Internal tool registry, policy check, target binding, and tool-run mediation. |
| RAG service | Internal ingestion and retrieval, plus portal-mediated knowledge query. |
| Events | Server-Sent Events for task, approval, tool-run, evidence, and system updates. |
| Health/readiness | Every first-party service exposes liveness and readiness checks. |
The baseline does not grant production-changing authority. It defines the API shape that later implementation work must follow.
Implementation Status
The canonical repository now includes contract-shaped HTTP stubs and tests for the first service boundaries.
Public-safe status:
| Service | Current status |
|---|---|
| Portal API | /api/v1 route stubs, liveness, readiness, metrics, and container healthcheck mode. |
| Agent orchestrator | Internal /internal/v1/orchestrator route stubs, liveness, readiness, and metrics. |
| MCP gateway | Internal /internal/v1/mcp route stubs, liveness, readiness, and metrics. |
| RAG ingestion | Internal /internal/v1/rag ingestion service with durable Postgres metadata, reviewed source object storage in MinIO, deterministic chunking, deterministic local embeddings, Qdrant indexing, vector-first query with keyword fallback, liveness, readiness, and metrics. |
| Deployment | Compose health checks added for first-party service containers. |
| Shared contracts | Request ID, task ID, error envelope, list envelope, and health/readiness helpers added for first-party service stubs. |
| Operational store | Postgres-backed task records, approval requests and decisions, evidence metadata reads, and task audit events; deployed smoke checks passed on ze-tower. |
| Internal clients | Portal API calls orchestrator, MCP gateway, and RAG service routes for readiness, task enqueue/resume/cancel, inventory targets, and knowledge query; deployed smoke checks passed on ze-tower. |
| Approval-wait workflow | Execution-seeking tasks can enter awaiting_approval, create durable approval requests, resume when approved, and block when denied without enabling production-changing authority; deployed smoke checks passed on ze-tower. |
| Operator portal MVP | Browser UI for readiness, task creation/list/detail, approval decisions, knowledge search citations, inventory targets, evidence lookup, and task events through the Go/Gin portal API; deployed smoke checks passed on ze-tower. |
| Operator chat console | Browser UI for chat-driven task initiation, RAG citations, proposed plan steps, task state, and approval/risk state through the portal API only. |
| LAN portal ingress | Single trusted-LAN ingress path for the operator portal; internal services remain Docker-internal. |
| RAG retrieval evaluation | First reviewed phase1-control-plane knowledge pack and local fixture verify expected citations before deployment. |
The RAG ingestion path is suitable for reviewed, non-secret fixtures and early knowledge packs. It now has a vector retrieval path through deterministic local embeddings and Qdrant, but retrieval quality should still be treated as a bootstrap baseline until a model-backed embedding service is selected and evaluated.
The first retrieval fixture does not call private infrastructure. It ingests reviewed markdown into the in-memory RAG store and checks expected source citations for public-safe control-plane questions.
These service increments do not implement real infrastructure actions.
The operator portal MVP does not add production-changing authority. It gives operators a browser surface for already-governed Phase 1 API workflows while preserving the rule that the frontend only talks to the portal API.
The operator chat console does not add production-changing authority. It creates tracked tasks, queries reviewed RAG, and returns planning envelopes with citations and authority notes. Execution-seeking work remains approval-gated and still requires later orchestrator, policy, MCP, and evidence controls before real target changes are possible.
The LAN ingress change does not expose internal services or target infrastructure. It only makes the existing portal ingress reachable from trusted LAN clients.
Knowledge Base Direction
The knowledge base should be built from reviewed source notes, not random chat memory.
Good source material includes:
- deployment procedures;
- incident notes;
- cluster inventories;
- codebase architecture notes;
- runbooks;
- hardware and network inventory;
- SOPs;
- backup and recovery procedures;
- validated command outputs with sensitive values removed.
GitHub and source systems remain authoritative. Qdrant is a rebuildable retrieval index, not the source of truth.
The first reviewed pack is a control-plane pack. Future packs should cover target infrastructure domains such as OpenShift/Kubernetes, hypervisors, network, facilities, and software platforms after public-safety and operational review.