Reliability & Automation

Backups Built For The Day You Are Already Compromised

Everyone has backups. The question worth asking is whether the thing that just encrypted your fleet can also reach them, and for most people the honest answer is yes.

Designed, built, and operateresticappend-only rest-serverBackblaze B2systemd timersSSH forced commandsPrometheus

Three copies, two media, one offsite, and the offsite copy is pulled by a machine the fleet cannot reach.
No host on the fleet holds a credential that can delete a backup.
Restores are drilled daily. The number I keep is when one last succeeded.

The problem

Ask an operations team whether they have backups and they will say yes, immediately, with confidence. Ask when somebody last restored from one and watch the room get quiet. A backup nobody has ever restored is not a backup. It is a hypothesis with a budget line.

The threat also moved. Ransomware crews stopped encrypting and leaving. Now the first thing they look for is the backup server, because they understand better than most operators do that it is the only thing standing between you and paying them.

So this is not built for a failed disk. Disks are easy. It is built for the morning you discover the fleet has belonged to somebody else for a week.

Assume the fleet is already theirs

Every host pushes its backups to an on-site repository running append-only. A host can write a new snapshot. A host cannot delete an old one, cannot rewrite one, and cannot prune. Not because it is configured politely not to, but because the credential it holds does not carry that verb.

This matters because of what actually happens in a ransomware incident. The attacker gets a host, finds the backup credentials sitting right there in the backup script, and uses them. That works everywhere, because the same credential usually reads, writes and deletes. Split the verbs and the stolen credential becomes an ability to add data to my backups, which is not a threat I lose sleep over.

The copy nothing on the fleet can reach

The offsite copy is pulled, never pushed. A machine that is not part of the fleet reaches in on a schedule, takes a copy, and hangs up. Nothing on the fleet holds a credential to that machine. Nothing on the fleet knows how to reach it. A compromised control plane cannot delete the offsite copy for the same reason you cannot delete a file on a laptop that is switched off in another building.

It works from behind a home connection with no fixed address and no inbound holes, because it only ever dials out. And what it is allowed to do at the far end is not a matter of trust or of policy. It is three lines of configuration:

# jump host: forward a port. That is the entire grant.
restrict,port-forwarding <key>

# backup server: reach one local port, and never get a shell.
restrict,port-forwarding,permitopen="localhost:<repo port>",command="/bin/false" <key>

# metrics host: run one receiver script, with no arguments of its own.
restrict,command="<metrics receiver>" <key>

# note what is absent: no interactive session, no agent forwarding,
# no X11, no arbitrary command. The offsite box cannot be used as a
# foothold into the fleet, because it was never given a way in.

The offsite machine's capability, written as SSH key restrictions on each host it touches. It can forward a port through the jump host. On the backup server it can reach exactly one local port and can never obtain a shell. On the metrics host it can run exactly one script and nothing else. This is the whole permission model, and it is auditable by reading it. A credential that cannot express an action is stronger than a credential that is asked not to perform one.

Backup topology
Fleet hosts	append only, cannot delete or prune
On-site repository	holds snapshots, reachable by the fleet
Offsite machine	pulls a copy on a schedule, unreachable from the fleet

The direction of the last arrow is the entire argument. Hosts can add a snapshot and cannot remove one, and the offsite copy is taken by a machine the fleet has no way to address.

A backup you have not restored is a rumour

Verification reads the data, not the index. It is entirely possible to hold a repository whose catalogue is perfect and whose contents are corrupt, and you find out on the worst day of your professional life. So the check re-reads actual data rather than confirming that the bookkeeping agrees with itself.

Restores are drilled rather than discussed. The number that matters is not how many backups ran. It is when a restore last succeeded, and that is the number I keep.

If it is not on the dashboard, it did not happen

After every offsite run the machine pushes its own metrics back into the same monitoring the rest of the fleet reports to, so the offsite copy is visible next to every other backup, with alert rules on staleness.

A backup system that fails silently is worse than none, because you plan around it. The failure mode I am designing against is not the backup breaking. It is the backup breaking quietly, in February, and nobody finding out until August.

Engineering decisions

The calls I made, and what each one cost.

Append-only at the destination, so a host can add but never remove.

The credential on a fleet host is the credential an attacker gets. Splitting write from delete means stealing it buys them the ability to give me more backups.

The offsite copy is pulled by a machine the fleet cannot reach.

Push means the fleet holds a credential to the offsite copy, and anything the fleet holds is in scope the moment the fleet is compromised. Reversing the direction removes the credential from the blast radius entirely.

Verify by reading the data, and drill the restore.

An index that validates proves the bookkeeping is consistent. It proves nothing about the bytes. The only honest test of a backup is a restore, so that is the test.

The offsite machine reports into the same monitoring as everything else.

An unmonitored backup is a belief. I would rather be paged in February about a stale offsite copy than discover it in August with a lawyer on the call.

More systems

Reliability & Automation

I'm looking for Incident Commander and SRE roles.

If your team is drowning in toil, alert noise, or incidents that never quite close. That is the work I do.

Get in touch