Wiped: how a full-disk failure became a same-day restore

A full-disk wipe hit a one-box agent fleet and the whole thing was back the same day. The doctrine that did it: back up only what cannot be re-downloaded, script the rebuild, and guard the one key that unlocks everything.

The box that runs my agent fleet came back from a full-disk wipe on a Saturday morning, and by the same evening every cron job, every memory store and every hand-written service was working again. Nothing about that sentence was luck. It was a doctrine, written down as one rule, and this piece hands you that rule plus the script that turned a disaster into a bad morning.

My fleet lives on one rented Hetzner server, the same box from the very first article on this site. On it run about nineteen agent profiles: one keeps the calendar and the shopping, one mends the infrastructure, one remembers things in a long-term memory store, and so on. Each profile has its own config, skills and memories. Call the keeper of the infrastructure Spanner, and call the memory store Archive, because that is what they do. On 5 September somebody hit the reset button on all of it: the disk was wiped clean.

Count what burns before you buy insurance

Most backup advice starts with copying everything. That is the expensive way to do it, and after the wipe I learned why it is also the wrong way: a survey of the box found that roughly 95 percent of it could be re-downloaded or regenerated, so backing it up would have been paying to store copies of things the internet already holds.

Spanner did the survey while the smoke was still clearing. System packages reinstall from a package manager. Python environments rebuild from a lockfile. The website repo, the game repo, the vault mirror: all already on GitHub. Even the tools in the local bin directory, like the GitHub CLI and rclone, are one download away from existing again. What genuinely burned fell into a short list. The memory databases that Archive keeps had no other home. A handful of tiny state files, the kind that record what a cron job last did, existed nowhere else. Five small helper scripts, hand-written glue between the fleet and a secrets vault, were committed to git but only if the git was actually pushed. And one encryption key, which we will get to, because it nearly earned its own paragraph of panic.

So the rule writes itself, and it is the one load-bearing idea in this piece: back up only what cannot be re-downloaded. Everything else you re-buy, re-clone or re-install on the day you need it.

Three layers, one for each kind of unknown

That rule still leaves a question: re-downloadable from where? The answer needs somewhere to stand, and it stands on three layers, each covering a different kind of surprise.

Layer one is Hetzner Backups: seven daily slots, priced at a flat 20 percent of what the server costs each month, so about 1.60 EUR on my machine. This layer covers the unknown unknowns, anything the survey failed to enumerate. One honest wrinkle: only I can switch it on, from Hetzner’s console, because the agents have no access to my hosting account. After the wipe I did exactly that, first thing.

Layer two is git. The fleet’s home directory auto-commits itself, and the hand-authored systemd units, which are service definitions for things like a browser session and a text-to-speech server, have no generator, so they are the files that must reach a remote. The lesson from the survey is not “git saves you”. It is sharper: verify the push state, because git that stays local is a diary, not a backup. The command that settles it is:

git rev-list --left-right --count origin/main...HEAD

Two numbers come back. If the second is zero, everything local is on the remote. When I ran it on the fleet’s home repo during the restore audit, it answered 0 6: six commits sitting only on the box. That is the trap in one line of output, and it is why the check is a command and not a feeling.

Layer three is R2, Cloudflare’s object storage, which is a big sorted vault in the sky with a free tier of 10 GB and no charge for pulling data out. Archive’s memory databases land there nightly, encrypted, and the small state files join them weekly.

The cost line for the whole scheme reads: about 1.60 EUR a month for the server snapshots, zero for R2 at this scale, and zero for git. The real bill is the evening of work to write the runbook, which pays for itself the first time the disk goes.

What the fire exposed

Three things broke during the restore, and each one teaches the same kind of lesson: the failure was never where the plan said it would be.

The first break was the encryption key. All of Archive’s R2 backups are encrypted with age, a small encryption tool, and one single key decrypts all of them. After a wipe the key file is gone; the master copy lives in a secrets vault, and only the owner can retrieve it. Lose that key and the nightly backups become noise. The doctrine now says it plainly: the encryption key is the single point of failure, it belongs to a human, and the first question in any restore is “where is the key”, asked before anything else.

The second break was a ghost. A service that had been serving web pages kept answering requests after its directory was wiped, because a running program lives in memory, not on disk. A stale process was returning errors from files that no longer existed, which is a maddening thing to debug at 6 a.m. The lesson: before you restart anything after a restore, know whether the service definition exists on disk. A process with no unit file behind it is a ghost, and ghosts die with the next reboot and take their config knowledge with them.

The third break was the sneakiest: things that are re-downloadable are not the same as things that are backed up, and the difference costs hours. The text-to-speech stack needed a git clone plus a model download plus a rebuilt Python environment. The CLIs needed pinned versions, because a fresh “latest” download is a compatibility experiment you did not ask for. Re-downloadable is a plan, not a state. The plan has to be written down or it is just optimism.

The recipe: one bash script, ten sections

All three lessons live in one place now, and the place is a plain bash script called fleet-bootstrap.sh, 428 lines, which Spanner committed while the restore was still running. No Ansible, no configuration-management framework, just a script, because the fleet is one host and the script’s sections map one to one onto the jobs. The rule of thumb we wrote down: revisit the fancy tools at three machines, not before.

The script is idempotent, which means safe to run twice: every section checks whether its work is already done and skips if so, so a half-finished restore just continues where it stopped. Here is the shape of it, compressed to the steps you would actually follow:

1. Survey first. For every repo and every tool, decide: re-downloadable or irreplaceable? The push-state check from the last section is how you audit git. Write the two lists down.
2. Give the irreplaceables a home off the box. Memory databases to encrypted R2 nightly, state files weekly, hand-written systemd units and helper scripts to git, the encryption key to a secrets vault only you can open.
3. Write the runbook as plain bash. Pin every version (the script pins the GitHub CLI, age and others), print no secret values ever, and never restart a service that is still answering on its port.
4. Guard every section with a skip-if-present check. This one habit is what makes the script a runbook instead of a one-shot installer: re-running is free, and partial restores resume cleanly.
5. Prove it without a fire. bash -n checks the syntax in a second. Run the sections on a healthy box and read the summary: every section prints PASS or SKIP, and any FAIL names itself.
6. When the day comes, restore in order. Server image if you have one, then the bootstrap script, then the R2 pull, then the re-clones, then an audit of every cron job. The order matters because the script needs the vault token from step 2 before it can fetch anything else.

The full script is the fleet’s own, written for one host and its specific services, so I have not published it as a copy-paste artifact. The six steps above are the reproducible core: a reader can write their own version in an evening, and the skip-guard pattern in step 4 is the part worth stealing verbatim.

Honest limits

This doctrine has passed one live fire, which is both its strongest evidence and its only evidence. It is built for one host; a second machine would still work but a third should trigger the rethink we already named. Hetzner’s layer depends on me, a human with a browser, and that dependency is deliberate but real. The age key remains one lost file away from turning Archive’s memories into noise, and the mitigation is a vault plus a 0600 cache file, not a second copy in a second country. And a restore that has been rehearsed only in a real emergency has not really been rehearsed; the next test is a scheduled drill, done quarterly, with no fire to excuse the failures.

Takeaway

The useful lesson is not that one fleet survived one wipe. It is the pattern: survey what cannot be re-downloaded, back up only that, script the rebuild so it is idempotent, and put the one key that unlocks everything in a human’s hands. The box was a fire; the doctrine is the reason it was a small one. Run the push-state check on your own repos tonight, because the six unpushed commits it found here were the closest this fleet came to losing anything. When the drill has run, that is the next piece.