AIOps
An operations org that runs itself, and knows when to stop.
I've spent fifteen years cutting toil and time to recovery for teams of people. This is the same job with AI taking the first shift. Automated responders detect, triage and fix incidents on their own, and thirty-four agents organized like a company handle the engineering, security and follow-up around them, on a production fleet of customer websites, hosting accounts, game sites and apps, including ones that hold tax and health records. I get paged for the decisions that should be human, not to wake up and read alerts.
- The first action on an incident doesn't wait for anyone to be paged, wake up and open a laptop.
- Round-the-clock coverage that would take four to five people to staff runs with no on-call rotation. I'm paged only when the machine can't safely fix it.
- Lessons from incidents ship as checks that re-run every fifteen minutes and go red on their own, so a repeat gets caught.
Why
I was the one who got woken up, and a lot of the time I wasn't the one who could fix it. The longest stretch of an outage was paging the DBA or whoever owned the pool, then waiting while they woke up, logged in and caught up.
At eBay I automated 90% of the manual work off a twelve-person team, and we held a 99.997% uptime SLA for eleven years. The goal hasn't changed since then. What changed is that a machine can now read an alert, form a view and act inside a boundary I drew, which used to need someone awake at four in the morning.
| One seat, covered around the clock | 4.2 people |
|---|---|
| The NOC I ran at New Frontier | 12 people |
| This fleet | 1 people |
The incident loop
Every phase of an incident, closed by the machine
Each stage is a separate system with one job and one thing it is never allowed to do. That second line is what makes it safe to leave running.
- 1
Detect
Network and host detection, uptime and content checks every couple of minutes, error rates against each site's own baseline, and trend alerts that warn days before a disk, memory or backup store runs out. On its first day the trend alerting flagged a server disk eleven days from full.
Never: Calls a fix done on a bare 200.
Deep dive - 2
Triage
An AI investigator answers one question per case: is it real, and by what mechanism.
Never: Gathers its own evidence or raises a severity.
Deep dive - 3
Diagnose
The responder collects evidence before it acts, and a committee of models reads the live terminals it's armed on.
Never: Types into a terminal I haven't armed it on.
Deep dive - 4
Mitigate
The security engine picks one numbered action from a list approved in advance, then clears ten gates. The outage responder works from a written playbook and gets one safe try.
Never: The security engine writes a command or names a machine.
Deep dive - 5
Resolve
Counts a recovery only when the service serves correct content again. One safe attempt, then it escalates with the evidence attached.
Never: Retries the same fix in a loop.
Deep dive - 6
Learn
Lessons from incidents become checks, more than eighty of them, that re-run every fifteen minutes and go red on their own.
Never: Waits for a person to remember to look.
Deep dive
The org
Thirty-four agents, organized like a company
Each role is a written definition with its own scope and personality, and one roster file holds the reporting lines. A manager can only assign work inside its own branch. A few lanes cross it on purpose, like the Incident Commander, who can task any responder during a declared incident while their managers' rounds pause.
Click any seat for its job description.
Owner
Decisions that should be human
CEO's staff
Report straight to the CEO and sit outside the departments, so they can work across all of them.
Security
IT operations and engineering
Engineering
Operations
Customers
The same structure also runs the business side. Both report to the CEO.
Growth: marketing, sales and customer success
Hacked-site recovery, its own business line
The full chart, the dispatcher and how the gaps were found: The Agent Org Chart & Dispatcher.
Dispatch
How a piece of work moves
Nobody has to notice a task is waiting, start a session for it or chase it when it stalls. That in-between work was the toil, and it's the part I automated first.
- 1
Work arrives
An order from me, a manager's delegation, or a watcher filing what it found, like a new error signature on one of the sites.
- 2
Eligibility check
The dispatcher only starts a task that came from me, from someone above the assignee in the chart, or through a lane written into code. Anything else waits for a person.
- 3
A session starts
Within a minute when a slot is free, in its own terminal, under its own memory cap, with the screen-reading committee watching it.
- 4
Supervised
Never started, crashed, went idle or ran too long all count as a failed attempt. Three of those and the task blocks with the reason.
- 5
Proof, then done
Every agent is told to paste the output that proves the work. A manager's next round reads the done note, re-queues what isn't finished, and escalates what is stuck.
- 6
Only decisions reach me
A question with labelled options, what each one does, and a recommendation. My answer goes back on the task and the next session starts from it.
The human line
What runs alone, and what waits for me
Fast and unattended only works with a clear rule about what the machine may not do. Part of it is enforced in code: who can start work, what the security engine may do, what a model may type into a terminal. The rest is written into each agent's role and backed by permission prompts only a person can answer.
Runs on its own
- First response to an outage, from a written playbook with one safe try
- Closing a false-positive security alert
- Containment chosen from actions approved in advance
- Routing, starting and supervising every task on the board
- Code fixes with tests on the non-regulated sites
- Drafting every customer reply, notice and post
Waits for me
- Anything a customer will read
- Anything that spends or refunds money
- Destructive security actions. Tested by having the engine try
- Deploys to the apps holding tax, bank and health records
- A client's DNS
- Approving what a live session asks permission to do
Cost and runaway loops
Agents can't argue with each other, and can't run up a bill
The failure everyone is worried about now is a swarm of agents negotiating with each other in a loop while the meter runs. That can't happen here, and the reasons are structural rather than a line in a prompt.
Agents can't pretend to be each other
Who an agent is comes from the sandbox the dispatcher started it in, not from what the agent says. A session that files work under any other name, its manager's or the CEO's, is refused, nothing is saved, and the attempt is logged off the box.
Approvals can't be forged
No agent session can answer a question as me. When I approve something, the approval is bound to the question it answers and works once. Identity the kernel enforces is next, then per-role keys in the vault.
Agents don't talk to each other
No agent can call, prompt or message another directly. Work moves only as a task on the board, and the dispatcher checks who filed it before it starts anything. The few lanes that cross the chart are written into code, and each hop is a new task under the same caps. There's no live back-and-forth for two agents to get stuck in.
Disagreement slows things down
When models judging a situation disagree, the response gets more careful, not louder. The terminal committee holds and types nothing unless a second panel agrees, and a dissent in the security engine can only pull the action down. No committee keeps meeting until someone gives in.
Managers work in rounds
A manager isn't a session left running all day. It starts when there is something to manage, reviews its branch, closes the round with a one-line report and exits. Rounds have their own cap on how many run at once and how long each may take.
Every session has a ceiling
Outside a declared incident, only a few sessions run at once, one per role. A session idle for half an hour is closed, and so is one past four hours, unless it's waiting on me at a permission prompt. A task that fails three times blocks with the reason instead of retrying forever.
One stop for everything, and no self-approval
One command stops every lane on the next pass, tasks, rounds and incidents, and can end running sessions too. The agents write code and run commands inside Claude Code's permission system, and anything it won't clear stops at a prompt. The dispatcher never answers those prompts. A person does.
My own caps, not a meter
The agents run on my own Claude subscription, not pay-per-call API keys, On top of the plan's limits I set two of my own: a daily cap on work sessions and one on manager rounds. When either runs out, routine work waits for tomorrow unless I raise the cap, and a declared incident still gets its responders.
Decisions
The calls that make it safe to leave running
The speed is the point. These are why it doesn't come at the cost of judgment.
The security engine picks from a list. It never writes the command.
If a hostile log line can steer the model, the worst it can do is make it choose a different action that was already approved. A target or a command is something the protocol can't express, so there's nothing for a validator to miss.
Unattended work flows down the chart.
A task becomes a prompt that runs on its own, so the dispatcher checks who filed it. Work flows down the reporting line, with three exceptions written into code: requests to the security team, a program lead's named team, and the incident commander during a declared incident. A confused agent can't hand its boss or a peer a prompt any other way.
Recovery means correct content, not a status code.
A 200 means something is listening. The outage I care about is the one where every dashboard is green and the customer sees the wrong page.
Disagreement makes it more careful.
The committee that reads live terminals is three readers on the same screen. When they disagree, nothing is typed unless a second panel of different models agrees, and that panel can only release an answer already on the safe list. In the security engine, a dissenting vote can only pull the response down a step, never up.
The org chart was benchmarked, not guessed.
Three AI reviewers, one per discipline, compared it with how ITIL shops, SRE teams and security organizations actually staff operations. Most gaps turned out to be duties nobody owned, which went into existing roles. Only seats with recurring work or a separation-of-duties reason were added.
Go deeper
Each piece, with the evidence
Every deep dive has the decisions I made, the ones I got wrong first, and the artifact that backs up the claim.
The Fleet Control Plane
The security operations, autonomous response, network detection, self-healing incident response and worker pool that run every other system here. Built by one operator, deliberately shaped as if a team were about to inherit it.
The Agent Org Chart & Dispatcher
Thirty-four AI agents arranged as a company: a CEO, heads of security, IT, engineering and marketing, an incident commander, and the specialists under them. A dispatcher starts a session for each task, runs manager review rounds, and enforces who is allowed to hand work to whom.
The Self-Healing NOC & War Room
An unattended first responder for production incidents. It works from a written list of actions it is allowed to take, it only counts a recovery when the service serves its own correct content, and when it cannot fix something it stops and wakes a person.
The AI SOC
The part of a security operation that decides whether an alert is real, and by what mechanism. AI does the reading a whole team would otherwise do, and it is never allowed to go collect its own evidence.
Autonomous Response
An engine that responds to security incidents on its own. Built so it can only ever choose from actions authorized in advance. It cannot name a machine or write a command, and anything destructive still needs a human.
Agent Identity
How 34 AI agents prove who they are before their work is trusted: identity read from the session's own sandbox, never from what the agent claims, with every refused impersonation logged to the security pipeline. Identity the kernel enforces comes next, then per-role keys in the secrets vault.
Invariants: Hunting the False Green
80 fleet invariants every 15 minutes and 10 more hourly, each written after an incident where a dashboard read green above something broken. The suite also fails on its own blind spots.
Secrets & Key Custody
How the fleet's OpenBao vault keeps per-tenant encryption keys off the app servers, unseals only with shares held off the machine, and is watched by checks that never mistake not running for passing.
The Terminal Brain
Browser SSH to fleet hosts with durable sessions, plus an AI committee that reads the screen when it goes quiet, grades the risk, and can only ever hand a suggestion to a human who presses a key.
The Fleet Worker Pool
A distributed job and inference fleet. Any box serves any application's queue, caller code runs sandboxed with nothing of its own to write to, and model routing reroutes itself as providers throttle and retire models underneath it.
NDR Rollout
A staged network-detection rollout across regulated hosts, where the finding that shaped the whole design came from testing the integration everyone assumes works. Before installing anything.