Reliability & Automation

The Fleet Control Plane

One person cannot watch a production fleet around the clock. So the machine takes the first action, and the interesting part is everything that stops it.

Designed, built, and operatePythonBashsystemdWazuhSuricataPrometheusAnsibleClaude agents

Covering one seat around the clock takes four to five people. This runs on one, and the first action does not wait for me to wake up.
Every postmortem finding ships as a check that re-proves itself on a schedule.
No change surprises the on-call: work is declared, and suppression is one host for one window.

The problem

Every honest conversation about automating operations starts in the same place: not what the machine can do, but what happens on the day it is confidently wrong at three in the morning while you are asleep.

Most of the work here is that day. The parts that act are small. The parts that refuse to act, that re-check their own assumptions, that stop and wake somebody with the evidence already gathered, are most of the code.

What it actually is

A detection plane that decides whether an alert is real and by what mechanism. A response engine whose model output is one number, an index into actions authorized in advance. Network detection across the hosts carrying regulated data. An unattended first responder that verifies recovery by content rather than by status code. A worker pool that runs untrusted code in a sandbox and routes model calls around providers as they rate limit and retire models. And a terminal with a committee reading over your shoulder that can only ever hand you a suggestion.

Each of those has its own write-up. What follows is the part they share, which is the part I would actually want to talk about.

The humans never left

This is not a system that replaced an operator. It is a system that changed what the operator does. The scarce thing at three in the morning is not hands, it is context: what is broken, what has already been tried, what state was preserved before anyone touched it.

So the machine's job is to arrive at that moment having already done the boring, perishable, irreversible-if-skipped work. Evidence preserved before any decision, because a deleted but running binary exists only as an open file handle and it will not survive a reboot you ordered in a panic. Options built and ranked. A case that reads like a briefing.

Then it either takes an action it was authorized to take, or it stops and pages a person who starts by deciding instead of digging.

People required for continuous coverage
One seat, covered around the clock	4.2 people
The NOC I ran at New Frontier	12 people
This fleet	1 people

Continuous coverage is arithmetic before it is technology. One seat, covered around the clock, is 4.2 people at forty hours a week, before holidays, illness, or anyone quitting. I have staffed that desk. This fleet does not have it.

A change protocol, because the automation has to know when you are working

The failure everybody hits with self-healing infrastructure is the responder fighting the operator. You restart a service on purpose, the automation sees a service that just went down, and now two of you are working on the same box with opposite intentions.

So work is declared. Opening a window suppresses paging and remediation for that host only, and only for as long as it lasts. It delays an alarm. It never silences one.

$ fleet change start --host <host> --what "restarting the portal after a config change"
  change window opened: chg-4812  (ttl 120s)

  ... do the work ...

$ fleet change done chg-4812 --note "portal healthy, verified serving its own content"
  chg-4812 closed. host re-arms after settle.

# while open:  paging and auto-remediation suppressed, that host only
# on close:    settle period, then normal arming resumes
# on expiry:   the window lapses and a real problem pages normally
# every window is logged, and shows on a public status banner

The protocol, in full. The important line is the second one: whoever made the change closes their own window. Auto-expiry exists, but it is a safety net for the case where you crashed, not the normal path. After a window closes the responder waits out a settle period before re-arming, so a service still warming up does not trip the thing you just told to stand down.

Built for a team I do not have

I am the only operator. That is a real limitation and pretending otherwise would be silly. But the shape of the thing is not a shape you get from working alone: escalation paths, declared change windows, written action allowlists, evidence handoff, invariants derived from actual postmortems rather than from imagination.

I built it the way I would want it handed to me. Partly because that is the discipline, and partly because a system only I can operate is a system that fails the week I get the flu.

Engineering decisions

The calls I made, and what each one cost.

The model never emits a command. It returns an index.

An allowlist still means receiving a value an attacker may have influenced and then checking it. An index into a list the control plane built means there is nothing to check, because there is nothing to tamper with.

Nothing stays automated unless its assumptions are continuously re-proved.

A corrective action that quietly stopped being true is worse than no automation, because you are still counting on it. Every invariant here is traceable to something that actually went wrong, and every one of them is re-asserted on a schedule rather than trusted.

Verify recovery by content, never by status code.

A 200 proves a process is listening. It does not prove the service is serving its own correct content, and the gap between those two is where an outage hides while every dashboard stays green.

Destructive actions are human only, structurally.

Not policy, not a flag someone could clear. Unanimous agreement from fully monitored voters choosing the most invasive option on a disposable host still returns page a human. Tested, not assumed.

More systems

Security & Response

I'm looking for Incident Commander and SRE roles.

If your team is drowning in toil, alert noise, or incidents that never quite close. That is the work I do.

Get in touch