Reliability & Automation

The Self-Healing NOC & War Room

A 200 means something is listening. It does not mean the site is up. That gap is where an outage sits quietly while every dashboard stays green.

Designed, built, and operateClaude (headless)PrometheusGatusGrafanaAnsiblesystemd

A staffed first-response rotation for nights and weekends is four to five people. This is none of them, and it answers in seconds.
A recovery counts only when the service serves its own correct content, never on a status code.
One safe attempt, then it escalates with the evidence already gathered.

The problem

Auto-remediation usually means a script that restarts something when a probe goes red. That script hides crash loops, fights your deploys, and one day, at the worst possible hour, restarts a database.

So this one gets invoked for exactly one incident at a time, with no shell to wander off into, carrying an incident-commander brief and a written list of what it may do. One rule sits at the top of that brief: do no harm beats restore service. A fix that could corrupt data or bury a security problem is worse than the outage it was called for.

The loop, always in this order

Stages before the first action
Staffed rotation	Detect, Page, Wake up, Open laptop, Gather context, First action
Unattended first response	Detect, First action, Verify, Escalate with evidence

Where the time actually goes before anything gets fixed. The struck stages exist only because a human has to be located first, and they are the largest term in the number everyone reports as mean time to recovery.

1. Verify the symptom yourself
Don't trust one signal. Re-probe. If it is already healthy, resolve it as a flap and take no action.
2. Gather a read-only evidence bundle
Host reachability, disk, container state and restart counts, metrics context, recent logs. Scrubbed, never echoing record contents.
3. Ask what changed
Last patch run, recent snapshots, time since deploy, and whether a declared maintenance window on this host just opened or closed. A blip inside an announced change is expected, not an incident.
4. Classify
Host unreachable, container down or crash-looping, resource exhaustion, TLS, application 5xx, or dependency failure.
5. Crash-loop guard
If it has already restarted many times recently, do not restart it again. That masks a real fault. Escalate.
6. Smallest safe reversible action
Snapshot first on any host that has snapshots. One attempt. Never loop-retry a failed action.
7. Re-verify by content, not status code
See below. This is the rule that took the longest to learn.

The false recovery

A site can return HTTP 200 while the reverse proxy serves the wrong backend. Stopping a container drops its router, and another site's router can claim the hostname, so the health check goes green while visitors get somebody else's homepage.

The responder therefore has to confirm that each site serves its own content before it resolves anything. A bare 200 is a false recovery and does not close an incident. When content is wrong but the container is healthy, the fix is a proxy reconverge, not another restart, and then re-verify again.

A worked example: the outage every dashboard called healthy

One stopped container, one health check going green, and visitors getting another site's homepage. This is the sequence the responder actually runs. The clock is the shape of the sequence rather than a measured trace.

t+0sA container stops. Its router leaves with it, and another site's router claims the hostname.
t+20sThe probe returns 200. Every dashboard is green. The site is serving somebody else's content.
t+25sThe responder re-probes rather than trusting one signal, and compares what came back against what this site is supposed to serve. Content mismatch, not a flap.
t+40sIt gathers a read-only evidence bundle: host reachability, disk, container state and restart counts, recent logs, and whether a declared change window just opened or closed.
t+55sCrash-loop guard. If this container had already restarted repeatedly, the answer is escalate, because another restart would only hide the fault.
t+1m10sSmallest safe reversible action: snapshot first, then one proxy reconverge. One attempt, never a retry loop.
t+1m30sRe-verify by content. The site has to serve its own page before anything is allowed to close.

Nobody was woken, and no status-code check anywhere would have noticed this. That is the failure I care about most: the monitoring is happy, and the customer is not.

The change protocol

Every human or agent change on a live host opens a declared window before touching anything, and closes it afterward. While a window is open, paging and auto-remediation are suppressed for that host only. After it closes the NOC waits a settle period before re-arming, so a service still warming up doesn't trip the responder.

It delays an alarm. It never silences one. A genuine problem after the window lapses pages normally. Every window is logged to a war-room change log and surfaced on a public status banner.

An unknown hostname is warned about and recorded anyway, never refused. A maintenance window that fails closed would leave the NOC paging during a change, which is worse than a typo in the log.

Postmortems that cannot quietly regress

Every incident writes a record. But an action item in a document is a control that lives in a document, which this fleet has repeatedly found is the same as no control at all.

So a lesson earns a permanent assertion if its violation would be silent, is security- or availability-relevant, and has actually happened here. Those run on a schedule and feed a board, and none were failing at the last check. That is the mechanism that turns “we should watch for that” into something that pages when it stops being true.

House rules, each of which is a postmortem

Assert the property, not the proxy.
A check that cannot fail is worse than no check. It occupies the slot a real one would take.
Deleting a file must never arm anything. Positive consent, never “absence of a disable flag.”
Fail closed for authorization; fail open for expiry. Never leave a permanent lockout.
Make failures visible where the operator is. A forced command died for six days because all of its evidence went to stderr, back down an SSH session, into a cron job that discarded it.
Test the self-healing; don't assume it. Delete the thing and watch it come back. A re-assert loop that looked correct had a short-circuit that never restored a missing rule. Found only by deleting one.
Verify before you claim. Run it and paste the output. “Should work” is not a result.

Engineering decisions

The calls I made, and what each one cost.

The responder escalates instead of retrying.

One safe attempt, then a human with an evidence bundle. A responder that retries is a responder that turns a five-minute outage into a corrupted database at 4am.

Suppression is per-host and time-boxed, never global.

A blanket mute is how an unrelated real outage gets swallowed by somebody else's deploy. The window names one host, and closing it is the change author's responsibility. Auto-expiry is only a crash safety net.

Write the standard you can actually meet at 3am, not the aspirational one.

You will be held to whichever one you wrote down.

Write findings as measurements, not verdicts.

“Detection was completely inert” is a self-criticism. “This rule produced zero events across four test variants; root cause was a watch bound to a symlink inode; fixed and re-tested, four of four now detected” is a fact, and it is the one that survives being read aloud by someone hostile.

More systems

Reliability & Automation

I'm looking for Incident Commander and SRE roles.

If your team is drowning in toil, alert noise, or incidents that never quite close. That is the work I do.

Get in touch