The monthly review at a mid-sized Ahmedabad logistics firm ran eleven minutes long, mostly on good news. An agent had been auto-closing the company’s phishing alerts for a quarter — a textbook AIOps for Security deployment, or so the slide said — and the queue had shrunk by four-fifths, with a “hours saved” figure that made the CFO smile. Then the newest ops lead, three weeks into the job and still asking junior questions, raised her hand: “If it closed four out of five, what did we stop seeing?” The room went quiet in a different way. Her question opens the road most deployments in this field never travel — the miss side, where the failure mode is silence — and this piece walks it end to end.
The Problem with AIOps for Security Success Stories
Every deployment story here gets told from the same chair: the one watching workload disappear. Automation share, hours saved, queue shrinkage — real numbers, all measuring the comfortable direction. None of them measures the opposite direction: genuine threats the machine passed over while everybody watched the savings. A closer that dismisses 95% of the queue might be catching nearly everything, or might be burying one genuine intrusion in a hundred. The slide cannot tell you which. That asymmetry — loud savings, silent misses — is the structural problem this diagnosis takes apart. Three questions, asked in order, would have found every issue her team later fixed:
- What is the worst case the trial’s sample actually supports?
- What could our own audit genuinely see — and what does it miss?
- Which categories hide inside the blended score?
Diagnosis One: The Trial Measured the Wrong Thing
Start with the field’s anchor trial, because it shows how good numbers can still fail a proper grade. Humans had labelled 1,240 emails in advance; 71 were genuinely dangerous. Two of those 71 slipped past the agent. Two misses out of seventy-one reads as 2.8% — tidy, presentable, almost good enough. Then the statistics speak: at a sample that size, the 95% worst case sits near 8.6%, and the strict gate careful teams require before trusting automatic closure is 2%. The trial fails despite its tidy surface, and the same trial’s other numbers show why the picture stays complicated: close calls were right 99.8% of the time, its own “this is genuine” verdicts carried only 86% precision, and 93.5% of analyst workload proved removable — real relief, real risk, simultaneously. Her question for any future vendor is the one worth stealing: “What is the worst case your sample supports?” Brochures carry averages; contracts should carry bounds.
Diagnosis Two: The Audit That Couldn’t See
The natural response — “fine, we’ll audit it ourselves” — runs into the second failure. Most teams audit by sampling history: pull a slice of closed items, check them, feel reassured. The arithmetic refuses to cooperate. Misses are rare events, and rare events hide from small samples: a spotless run of n cases still tolerates roughly 3 errors per n underneath. A sample of about ninety items drawn from a quarter’s ~1,845 closures, coming back totally clean, still permits a worst case near 38%. Ninety spotless checks and you know almost nothing. This is why her audit plan replaced history sampling with planted evidence: about fifty canary cases every quarter, each one a genuine threat crafted by her own team, slipped into the flow where nobody could route around them, under a three-strike rule — the third canary the agent closes ends its auto-close privileges that day. History tells you about last quarter. Canaries tell you about this one.
Diagnosis Three: One Score Hiding One Failure
The third failure is the blended score, and it nearly fooled this team too. The vendor’s dashboard showed one accuracy figure, aggregated across everything. Aggregation is where category disasters go to disappear: the documented pattern left cloud-identity alerts missed at a 42.3% rate while phishing sat at a comfy 9.1%. Imagine that split landing in a logistics firm — phishing is annoying, but one compromised cloud identity can read the entire shipment ledger. Her standing order after that: category tables in every monthly review, weighted by damage rather than by volume, and no blended figure accepted as a summary of anything.
The Fix for AIOps for Security Misses
Notice that none of the three diagnoses argues against automation; each argues for measuring the direction the dashboard ignores. The fix that survived her first quarter fits on one page. Grade trust per action type, never per product: watching is free; recommending needs labelled evidence — 200-plus graded alerts carrying 30-plus genuine cases; auto-closing needs demonstrated misses capped at 2%, counted over 150-plus true cases, with canaries and an untouched holdout; reversible actions need wrong moves held below 1% across 300-plus near-acting verdicts with rollback rehearsed; and the irreversible rung stays human permanently, because accountability cannot transfer to software any more than blame can. Keep humans reading the machinery’s verdicts — MITRE’s guidance is plain that a person must stay in that loop. And keep the agent’s inputs in mind as a separate risk: it reads attacker-written text for a living, indirect prompt injection has been a recognised problem since 2023, and the NCSC’s December 2025 note argues the weakness does not behave like SQL injection at all, which is why least privilege and deterministic guards appear on her one-pager too. She also mapped the team onto a maturity path she found in the reference material: ad hoc today, measured once blind shadow scoring ran, governed once the register was signed, adversarial once purple-team canaries and incident-level recall checks arrived, and eventually — if the discipline held — actually learning from every incident.
When the inevitable benchmark claim arrived in the next vendor pitch, it met the same suspicion. ITBench handed agents compliance-and-security scenarios and counted a 25.2% resolution rate. CyberSOCEval’s landed between roughly 23% and 34% when agents worked malware-analysis tasks, and between 43% and 53% when the tasks were threat-intel reasoning, while blind guessing scored 0.63% and 1.7%. Better than chance, both times. Not close to finished, either.
AIOps for Security, Judged From the Quiet Side
Back to the meeting in Ahmedabad. The CFO’s smile returned once the numbers were reframed: hours saved remained true, and now sat beside a miss bound the firm itself had measured, canary results the firm itself controlled, and a category table nobody could smooth over. That is what a healthy deployment review looks like — the same slide, twice as honest, and eleven minutes that finally deserved the room’s attention. Teams wanting the wider operational context behind these tools will find a step-by-step guide to building a DevOps career useful background on how such platforms get run, and the general habit of checking before trusting transfers from everywhere, including first trips — first-time visitors to Gujarat plan by asking locals what the brochures omit, which is precisely the skill her question displayed. Even gadget buyers practise it, as anyone who starts with comparing drone features before finding experts already knows. The definitions, gates, and arithmetic referenced throughout are collected in the Ops Atlas material on AIOps for Security — the same source her team taped, metaphorically, to the review-room wall. Misses do not announce themselves. Only measurement makes them speak, and measurement is a habit, not a purchase.
