AI-assisted incident investigation, humans approve

During a bad on-call shift, the hardest part of an incident is rarely the fix; it is knowing what changed and what the blast radius is. A payments-adjacent team ran an experiment: could an assistant make that investigation fast, while keeping every action human-approved? The answer was yes, but only because the boundary was structural, not aspirational.

The problem: investigation is the slow, error-prone part

When an incident hit, the team scattered: one person checked recent deploys, another pulled dashboards, a third grepped logs. Reconnaissance was the bottleneck. By the time the picture was assembled, the window for a safe intervention had often passed, and the reconstruction was driven by memory.

The assistant's job is to assemble the picture faster. It is never the person who presses the button.

The approach: read the graph, propose, never execute

The control plane holds the inventory, topology, change history, and policy results, everything an investigation needs. The AI assistant reads that graph and proposes: here is what changed around the alert, here are the affected services, here is a candidate next step. Every proposed step is presented for approval, and the platform executes only what a human approves inside the policy and audit model.

alert ──▶ assistant reads inventory + change history        └─ proposes affected services + next stephuman review ──▶ approved? ──▶ policy check ──▶ apply                      └── rejected ──▶ recordedall steps ──▶ audit trail (proposal + approver + outcome)

Implementation steps

• Wired the assistant to the graph (inventory, topology, change history, policy).
• Scoped it to investigation and proposal: no execution path.
• Defined the approval loop so every proposed step is human-approved before apply.
• Enabled audit capture of proposal, approver, and outcome for each step.
• Drilled the loop on a simulated incident: faster recon, same human accountability.

Guardrails: the boundary is the product

The rule that made the experiment work was simple and non-negotiable: the assistant proposes, humans decide. It can read anything in the graph, but it cannot apply anything on its own. Every action stays scoped to approved capabilities, subject to policy and human review, and is recorded in the audit trail.

Lessons learned

Investigation is the right place for AI in operations. The graph already contains the answer to 'what changed and what does it affect'; the assistant's only skill is assembling it faster than a human can click through consoles.

Second, make the boundary structural. If the assistant had any path to execution, the whole trust model collapses. Keeping it investigation-only is not a preference; it is the design.

Third, drill before you need it. Run the loop on a simulated incident so the human review habit is muscle memory when a real one hits. The audit trail from the drill doubles as a training artifact.