Reliability & Automation

Invariants: Hunting the False Green

Every serious incident here had the same shape: a signal looked healthy while the thing behind it was dead. So the fleet asserts what must stay true, and the suite checks its own coverage.

Designed, built, and operateBashPythonPrometheuspromtoolnode_exporter textfile collectorntfycron

80 fleet invariants run every 15 minutes and 10 response-pipeline invariants run every hour.
If a check dies before it reports, the run marks itself failed.
A documented control that nothing enforces turns the suite red, and so does an enforced one nobody documented.
Every alerting rule is swept for metrics nobody emits.
33 promtool test files run five times a day against the live rules.
Every post-incident action item has to name the check that covers it, an open task, or a written reason it can't have one.

The problem

A green dashboard above a dead service is the most expensive lie in operations: it tells the people who would have caught the outage that there's nothing to look at.

This fleet has been bitten by it repeatedly. A backup exporter was killed by the OOM killer, and the last numbers it wrote were served as current forever. A wedged package lock reported zero pending updates for 38 hours. A security feed stayed fresh for 67 hours with nothing in it.

Documentation doesn't catch that, because it's written after the fact and then rots. So a control here is something that runs and can fail, and every check came from an incident or a near miss.

Six days of "awaiting authorization"

A service account on the control plane took pushes from my own machines through an SSH forced command, a command bound to a key in that account's key file. It fed the never-block allowlist, the addresses the armed containment lane must never block.

A correct hardening change breaks a control nobody could see

Times are relative to the commit, from the post-incident review.

T minus 5 minThe last successful write to the allowlist register.
TA hardening commit restricts the control-plane directory to root. Its pre-flight confirmed no non-root systemd unit or cron job used it. Both were true.
Every 30 minSSH authentication keeps succeeding. The forced command then fails with permission denied: the account can't traverse the directory.
Same runThe client matches "Permission denied" against its enrollment case and logs "awaiting authorization", so for six days it reports a key problem that doesn't exist. The real error goes back down the SSH session into a client cron job that discards it. The server keeps no trace.
About T plus 2 hThe last dynamic entries age out. The containment lane is now consulting only the static half of the allowlist.
T plus 5 d 20 hFound by a six-team security review. No alert ever fired.

No harm landed: both containment audit trails showed zero actions against any owner or fleet address in the window. The pre-flight had missed a third kind of invoker, one that appears in no unit, no crontab and no process list. The first invariant in the suite now asserts every 15 minutes that every forced-command target is executable by the account that calls it.

How a check earns its slot

An invariant earns a slot when its violation is silent, security-relevant, and has happened here. Each checks the property, not the package. "Is the brute-force guard installed" is a checkbox. "Is its jail bound to the port the world can reach" is the question, and it came from finding 220 permanently banned addresses rejected on a port nobody could reach.

Runs overlap, so the loser of the lock skips instead of racing the winner's write, and a crash leaves the previous verdict rather than an empty file that reads as fresh. A hung sweep is paged by a separate freshness watcher.

Pages are edge-triggered, and a clear is announced at minimum priority. Delivery is an invariant too: recent real traffic on the alert channel counts as proof that pages arrive, and a low-priority canary goes out only when the channel has been silent. A scheduled canary once made up 54% of all alert traffic, and a monitor that's the loudest thing on its own channel is a bad monitor.

Monitoring the monitoring

A check that crashes before recording a verdict would vanish, and the run would still report zero failures. So the run compares the number of verdicts with the number of checks dispatched and fails itself when they differ.

The page explaining the invariants once documented eight while the script asserted more than forty, and one row described an access grant that had been removed. The page warning about documented controls that don't exist was carrying one. Now a parity check compares the page with both suites in both directions and fails on drift, including for itself.

Alert rules get the same treatment. A rule like x == 0 evaluates to nothing when x has no series, so it looks armed and never fires. A sweep compares every alerting rule with every metric name in Prometheus and says whether the rule is wrong or the host is behind on a deploy. Early versions read PromQL syntax as metric names and called 93 of 199 rules broken when the real number was one. A checker that cries wolf gets muted, so it carries a selftest of every misparse it ever shipped.

Another check asserts that every rule file matches git and that Prometheus has loaded every group, because a file can match git byte for byte and still be inert. The promtool tests once ran one file of four and printed pass, so the job now fails if it graded fewer tests than it found.

The guards have regression tests too: 14 files that feed the checks broken inputs and confirm they go red. One pins the fix for a review date compared as a string, where a typo like "soon" sorted above every real date and bought a permanent exception.

rule_metric_sweep_evaluated 1
rule_metric_sweep_dead_rules 2
rule_metric_sweep_dead_orphan 2
rule_metric_sweep_dead_undeployed 0
rule_metric_sweep_rules_total 341

The rule sweep on October 20. Two rules point at a metric nothing emits, and the sweep blames the rule, not a host behind on deploys. When it can't reach Prometheus it publishes evaluated 0 rather than a clean zero, and an alert fires after 48 hours without a usable verdict.

From thresholds to trends

Fixed percentages say nothing about speed. A disk at 70% gaining 10 GB an hour is an outage tonight, and a disk at 86% that hasn't moved in a month isn't. The capacity rules project the trend and fire on time to full, from 24 hours out to 60 days. They were backtested against 14 days of real data before deploy, and the three disks the backtest showed passing 80% within 60 days became capacity tasks.

Web errors work the same way. Every five minutes an exporter turns the access logs on the web hosts and client portals into request and 5xx counters. Only status, site and duration cross the wire, so no address, path or user ID leaves the regulated portals. An alert fires when a site's 15-minute 5xx share is over 5% and three times its own 7-day share.

What it doesn't catch yet

Pages are edge-triggered, so a second failure that arrives while the suite is already red is quieter than the first. That trade-off is deliberate, and closing it is on the list.

Unknown is never a pass, but it isn't a verdict either: a sleeping home-desktop worker is recorded as skipped. One host can't be shelled into by design, so its drift is inferred from which metrics it publishes rather than read directly.

Two action items from that incident are still open tasks. Until this month nothing would have noticed the log was never built. A new check requires every post-incident action item to name the invariant that covers it, and backfilling it against old reviews is how this surfaced.

Engineering decisions

The calls I made, and what each one cost.

Assert the property, not the artifact.

Every false green here measured something next to the truth: a file's age, a running process, an auth log line. Checking the property costs more code and access per check, and it's the only kind that would have caught the incidents above.

Treat false positives as bugs as serious as false negatives.

A checker that was wrong one run in five failed 354 times over 26 days and was never wired to anything, because nobody trusted it. Precise suppression takes longer to build, but a noisy check gets muted, and a muted check is as good as none.

Page on edges, not levels.

Re-paging every 15 minutes for a known failure trains people to ignore the pager. The cost is a quieter second failure, and that gap is stated here instead of hidden.

Make the suite fail on its own blind spots.

A dead check, a doc row with no assertion, an alert on a missing metric, a test that never ran: each reads as green unless something counts it. Counting makes the suite noisier about itself, and that's what lets me trust it when it's quiet.

More systems

Reliability & Automation

I'm looking for Incident Commander, SRE and AIOps roles.

If your team is drowning in toil, alert noise, or incidents that never quite close. That is the work I do.

Get in touch