AI for Data Centre Operations

Data centres face unique operational challenges at scale: thousands of servers, complex dependencies between services, regulatory compliance requirements, and SLA commitments that demand 99.99% uptime. Traditional monitoring tools generate enormous alert volumes — most of which require human triage before action can be taken.

AIOP at data centre scale: Prometheus federation across multiple datacentres, n8n cluster mode for parallel workflow execution, and multi-agent orchestration that coordinates across 6 specialist agents simultaneously — handling 1,000+ servers without bottlenecks.

Predictive Maintenance for Data Centres

AIOP's Monitoring Agent runs daily Prometheus trajectory analysis using predict_linear() queries to identify impending failures 48 hours before they manifest. Three-tier warning system:

  • Watch (30-day trajectory) — logged and noted, no immediate action required
  • Warning (7-day trajectory) — Slack notification, scheduled remediation recommended
  • Imminent Failure (48-hour trajectory) — urgent page, immediate action required

Self-Healing Infrastructure at Scale

AIOP's five self-healing playbooks operate 24/7 without human involvement for L1 and L2 scenarios. Each playbook implements a verify-before-act pattern (confirming the problem is real), acts-then-verifies (confirming the fix worked), and escalates with a pre-prepared diagnostic report if auto-remediation fails after two attempts.

Compliance and Audit

Every AI action is logged permanently to the agent_actions table with action level, command, result, and — for Level 3 actions — the approving engineer's identity and timestamp. The REVOKE DELETE constraint applied in Episode 18 makes this an immutable compliance record.