Certificates Without Surprises

I was asked for "auto-renew anything under 29 days." A three-reviewer committee measured the fleet and turned it down. Every renewer here already triggers around 29 or 30 days, so at 29 you can't tell a renewal in progress from a broken one. Two daily actuators would also share one Let's Encrypt bucket: five duplicate certificates per rolling week, used up in about two and a half days by two force-renewals a day. That was the August 4 mail outage. The same certificate carried the alerting system's outbound mail, so 174 notifications were dropped over about seven hours and nothing reported it.

So one script issues. Every morning it renews anything under 25 days, warns at 21 and pages at 14, holding a lock the guardian shares. Success is judged by re-reading certificates after the action, never by its exit code.

Right after issuing, in the same run and under the same lock, a guardian takes inventory. It was built to kill this false green.

On August 14 a client site was 19 days from expiry while AutoSSL logged "Ready for Renewal" every three hours for 71 days. The expiry alert was working, and it said the wrong thing. The real cause was that Let's Encrypt couldn't reach the server through a firewall that only admits Cloudflare.

So an auto-renewed certificate under 25 days is now reported as a stalled renewer, separately from expiring. The headline minimum also leaves out certificates that serve no traffic, like stale lineages left behind by a migration, because a minimum set by a dead certificate is wrong in both directions.

A file copy that succeeds while the service holds the old certificate in memory is the August 8 failure. So the guardian compares the SHA-256 fingerprint of the file on disk with the one served over a live handshake. Mail is checked on all three TLS ports, because the daemons reload independently. The TURN relay caches its certificate at start with no hot reload, so a file-only check there would pass for about 60 days over a stale wire.

The check gates on whether it parsed a certificate, never on the exit code. Origins that require client certificates abort the handshake and exit non-zero while perfectly healthy.

Key material is backed up nightly as its own snapshot, and every Sunday a self-test restores it: keys parse, each key matches its certificate, the backed-up mail certificate is the one being served. In August a negative test caught the test itself: with both files missing, it hashed two empty inputs, got matching hashes, and passed. It now parses before it compares.

Wire proof covers the mail ports and the TURN relay. Everything else is checked at the source, not fingerprinted off the wire. The guardian can re-sync a drifted copy, but the scheduled run is report-only, so drift gets a person.

Cloudflare Origin CA certificates last 15 years and have no renewal mechanism. Cloudflare shows the private key once, so they're flagged for manual reissue. The secrets vault's own certificate is on a list no automation may touch: renewing it restarts the vault, the vault comes back sealed, and the regulated portals pause their encrypted features until a person enters three of the five key shares.