Security & Response

Secrets & Key Custody

One vault holds every regulated tenant's encryption keys. The checks around it count what they can't see and refuse deletes they can't prove are safe.

Designed, built, and operateOpenBaoTransit envelope encryptionShamir unsealPythonBashPrometheusWazuhrestic

Steal a tenant server's database, disk or backup and the sensitive fields and signed documents in it are ciphertext. The keys that open it live in the vault, on a different machine.
A sealed vault doesn't take a site down. App secrets are rendered at deploy, so a restart never needs the vault.
A rotated audit log can be deleted only after a long-term backup snapshot provably holds that exact file.
36,350 decrypts of the same 159 records in four hours was staff pages polling, not a breach. The read-out detector counts records nobody has opened in 30 days instead.

The problem

Every regulated tenant's sensitive fields and signed documents are envelope-encrypted with a key in one OpenBao vault, off the tenant's server. Steal a tenant's database or backup and those fields are ciphertext. That moves the whole risk onto one machine.

So every check around the vault follows one rule: a check that can't run must never look like a pass.

Keys off the box, and out of the restart path

Each regulated tenant gets its own transit mount, and each service in it gets its own key and AppRole. A stolen portal credential can't use the e-sign key or reach another tenant. A new tenant is one idempotent script.

App secrets are rendered from the vault to each host's env file at 0600 when I deploy or rotate, and the running app reads the file. So the vault sits in the deploy path, not the restart path. If it's sealed, sites keep running and restart from the last rendered file. Only deploys and rotations wait.

The regulated portals are the deliberate exception: their per-record keys need a live call, so a sealed vault pauses documents, e-sign and encrypted tax fields. Those fail closed while login and the rest of the app stay up.

Three of five, held off the machine

The vault comes up sealed after every reboot and unseals with three of five Shamir shares, none of them on the machine. I type them at a prompt that stores nothing. Writing them to disk for auto-unseal would put the key that protects the keys right next to the keys.

An unattended reboot would leave encrypted features paused until I'm there, so that machine reboots in attended windows.

Counting admin tokens without being able to use one

One admin credential can decrypt every tenant's data, and the posture assumes exactly one exists, held off the machine. In August that was briefly false.

bao_admin_tokens_total 1
bao_admin_tokens_unexpected 0
bao_admin_tokens_unevaluated 0
bao_admin_tokens_newly_blind 0
bao_admin_tokens_blind_accepted 0
bao_admin_census_evaluated 1
bao_admin_census_token_ttl_hours 720

The census as it reads today. A clean reading needs all of it: one admin token, nothing unexpected, and the uninspectable count equal to the accepted count.

How a second year-long admin token appeared by accident

A careful rotation still produced a credential nobody intended.

RotateI minted a new admin token and kept the old one alive for a few minutes so a mistake couldn't lock me out.
Raise the ceilingI lifted the token mount's maximum lifetime past its 32-day cap so the new token could live longer.
Side effectRaising a mount ceiling also extends existing renewable tokens. The old one had 25 days left. Now it had 358, and nothing was counting.
CensusI wrote a census that runs hourly, looks up every token accessor and pages if an admin-class token isn't in the approved set.

Within the week the census read exactly one live admin token and zero unexpected. It reads the same this morning.

What the census does with what it can't see

The census never reads a token. It works on accessors, which describe a token and can't log in as it. Its credential can list and look up accessors, and I tested the denials: no restore, no token create or revoke, no decrypt, no key-value reads. That's why cron can hold it.

An accessor the census couldn't look up used to be skipped, which published a clean bill covering a token it had never examined. Now it's counted. I can review and accept one, and the alert compares the uninspectable count with the accepted count, so a known blind spot stays quiet and a new one pages.

It also won't report zero. One flag in the wrong order makes every lookup fail and the census announce that no admin tokens exist, so zero is published as unknown. And when its own credential expired in September and it went blind for a day, I made each successful run renew it and published its remaining life, which alerts a week ahead.

Audit logs you can only delete once they're provably kept

The vault's audit log is kept for a year, through a monthly long-term backup snapshot that is never pruned. In October, 48 rotated files, about 14 GB, were cleared off local disk. They're safe in those snapshots, but nothing alerted on a missed monthly snapshot, and nothing stopped a later cleanup from deleting a file no snapshot held yet.

The guard is now the only sanctioned way to delete a rotated audit file. It allows a delete only past a 35-day floor and only when a long-term snapshot already holds that exact file: same size and same modification time to the nanosecond. Names don't count, because log rotation renumbers every file daily. Size and nanosecond mtime survive a rename and don't survive a content change. If the backup repository doesn't answer, every delete is refused.

A guard only stops deletes that go through it, so twice a day it also compares the local files with the last run. A file that vanished while no snapshot held it raises an alert that stays red until someone reviews it.

Detecting a read-out, not a busy day

The obvious read-out detector counts decrypt calls. One September day a tenant's portal made 36,350 decrypts in four hours, all normal: staff pages polling every ten seconds and decrypting the same 159 records. A volume alert would page on that, and an attacker could hide under it.

A read-out has the opposite shape: many different records, most untouched for weeks. So a scheduled job counts, per tenant per day, the distinct ciphertexts decrypted that weren't decrypted in the previous 30 days and weren't just minted by a write. OpenBao HMACs request values in its audit log, so the job compares HMACs and never handles a ciphertext or plaintext. Padding with repeats doesn't lower the count.

Measured over the audit history, a normal day for every tenant sits far below its threshold. The result goes to Wazuh and pages once per tenant per day.

Backups checked where I'd restore from

The vault also holds an e-signature signing key, and losing it would make every signature ever made with it unverifiable. It's snapshotted daily, its checksums are verified before the file is published, and it goes offsite with the nightly backup. The job's AppRole can take a snapshot and is denied restore.

Every Monday a drill pulls the newest snapshot back out of the offsite copy and checks the archive, its checksums, its raft index and its age. The last run passed on a same-day snapshot.

What it doesn't cover yet

The read-out detector is tuned for bulk reads of records nobody has opened lately, so it isn't a complete answer to every kind of read. The baselines were also measured off-season, and a CPA opening last year's files in tax season will look new, so they get re-measured before then.

The restore isn't fully rehearsed. A snapshot restored into a throwaway cluster sealed immediately, because the root key inside is encrypted with the original cluster's shares. So a full restore needs a person with three shares. What I haven't run is unsealing a restored vault with the real shares and decrypting production data with it. That's a 20-minute drill with me in the room, and it's still open.

Engineering decisions

The calls I made, and what each one cost.

Render secrets at deploy instead of fetching them at runtime.

A single-node vault is fine only if its outage can't take sites down. The price: a rotated secret waits for the next deploy.

Manual three-of-five unseal, no auto-unseal.

Auto-unseal puts the unsealing key where a machine can reach it. I accepted that an unattended reboot pauses encrypted features until I'm there.

Count what the census can't see, and accept blind spots by name.

Skipping an unreadable token gave a clean bill for a token nobody had examined, and alerting on every blind spot fires forever. Comparing against an accepted set keeps a reviewed one quiet and makes a new one page.

Alert on distinct new records, not on decrypt volume.

Volume pages on normal polling and gives an attacker cover. Distinct new records are harder to pad, with a known gap on records staff open daily.

More systems

Reliability & Automation

I'm looking for Incident Commander, SRE and AIOps roles.

If your team is drowning in toil, alert noise, or incidents that never quite close. That is the work I do.

Get in touch