A Backup Isn't a Backup Until Something Tries to Restore It

I "had backups." Nightly cron, gzipped dumps, even an offsite copy. If you'd asked me, I'd have said the word with a straight face.

Then I actually audited what the rotation covered, and the honest inventory looked like this:

• One Postgres database was backed up nightly. The cluster had four. The other three — analytics history, a webshop with real customers — existed in exactly one place.
• Every image and file in object storage: one place.
• Redis: one place, if you count "the volume it lives on," which you shouldn't.
• The encryption key that guards the backups: two copies, both on the machine being backed up. A poetic single point of failure.
• And nothing, ever, had tried to restore any of it since the day it was written.

None of these things page you. A backup rotation with holes in it behaves exactly like a complete one, right up until the day it doesn't. This is the guide-shaped version of the afternoon I spent closing those gaps: the patterns that generalize, the snippets that do the work, and the two gotchas that bit me on the way.

Why put backups in a git repo?

For small, precious data — databases, not media libraries — a private git repo on a different host is a genuinely great backup target:

• git push is your offsite transport. No rclone config, no S3 lifecycle rules. If the machine dies, the repo lives.
• History is point-in-time recovery for free. A 7-file day-of-week rotation (dump-1.gz … dump-7.gz) keeps a week of versions on disk, and every nightly push preserves that night forever. git log -- dump-3.gz, git show <commit>:dump-3.gz > restore.gz, done.
• Unchanged files cost nothing. Git stores blobs by content hash, so a mirror of mostly-static files adds almost nothing per night.

The nightly push script is deliberately boring — and the commit and push are separate statements on purpose, so a failure surfaces in the log instead of vanishing into a pipe:

git add -A

if git diff --cached --quiet; then
  echo "[$(date -Is)] no changes to push"
  exit 0
fi

git commit -m "backup $(date -u +%F)"
git push
git log origin/master --oneline -1
echo "[$(date -Is)] pushed"

The size caveat is real: this works because my whole stateful world is tens of megabytes. If a nightly artifact reaches tens of MB per night, move it to object storage and keep git for the small stuff.

Why encrypt before committing?

Because the repo is offsite, and offsite means "on someone else's computer." Anything with PII, sessions, tokens, or customer data gets encrypted before it enters the repo:

pg_dumpall -U postgres --clean --if-exists \
  | gzip \
  | openssl enc -aes-256-cbc -pbkdf2 -iter 200000 -salt \
      -pass file:"$KEY" -out "postgres-all-$(date +%u).sql.gz.enc"

Symmetric OpenSSL with a key file is unfashionable and completely fine for this: one machine encrypts, the same operator decrypts, no key exchange problem exists.

Two lessons from my own rotation:

Dump the cluster, not the database. My original cron did pg_dump lans_tools — the one database that existed when I wrote it. Two more databases joined the cluster over the following months, and the cron neither knew nor cared. pg_dumpall picks up every database automatically, including the ones you haven't created yet. Backup code that requires you to remember to update it is backup code that's already out of date.

Reclassify when the data changes. That original single-db dump was deliberately unencrypted — aggregate stats, no PII, nothing to protect. The moment the dump became the whole cluster, it contained customer emails and orders, so the plain dump became an encrypted one. The sensitivity of a backup is the sensitivity of the most sensitive thing in it, re-evaluated every time the "in it" changes.

And the key: it must not live only on the machine it protects, or your encrypted offsite history and your VPS share a fate. Mine now has a third copy in a password manager. That's the one step no script can do for you.

The gotcha: salted encryption churns your repo

Here's the trap with encrypted files in git. Salted encryption is designed to produce different output every run — same input, brand-new bytes. Encrypt the same unchanged 1 MB archive nightly and git faithfully stores 365 brand-new blobs a year, for data that never changed. Your "backups of nothing happening" quietly outgrow the data.

The fix is to hash the plaintext and only re-encrypt when the hash moves:

hash=$(cd "$STAGE" && find . -type f -print0 | sort -z \
  | xargs -0 sha256sum | sha256sum | cut -d' ' -f1)

if [ ! -f "$DEST/.private.sha256" ] || [ "$hash" != "$(cat "$DEST/.private.sha256")" ]; then
  tar -C "$STAGE" -cf - . \
    | openssl enc -aes-256-cbc -pbkdf2 -iter 200000 -salt \
        -pass file:"$KEY" -out "$DEST/private.tar.enc.tmp"
  mv "$DEST/private.tar.enc.tmp" "$DEST/private.tar.enc"
  echo "$hash" > "$DEST/.private.sha256"
fi

Idle nights now cost zero repo growth. (The find | sort -z | sha256sum dance instead of hashing the tar: tar embeds timestamps, so the archive itself isn't a stable fingerprint of its contents.)

For public object-storage buckets — site assets, images — skip the encryption entirely and mc mirror them into the repo as plain files. Git's content-addressing does the change detection for you.

What does a restore drill actually check?

This is the part that turns "we write files nightly" into "we have backups." A quarterly cron decrypts and verifies every backup type, and each type gets two checks:

Freshness. Is the newest file younger than ~28 hours? This one check is secretly the most valuable in the script, because it converts "the backup cron died three months ago" — the classic silent failure — into an email. The drill isn't just testing restores; it's a watchdog for the whole rotation.

Restorability. Actually decrypt it, actually unpack it, and check something structural, not just "the file exists and is non-empty":

• The Postgres dump: decrypts, gunzips, and contains the expected number of PostgreSQL database dump complete markers — a truncated dump fails this even though it's a perfectly valid gzip.
• SQLite databases: PRAGMA integrity_check via three lines of Python, plus a table count.
• Redis snapshots: the decrypted RDB starts with the REDIS magic bytes.
• The object-store mirror: per-bucket object counts, local vs. remote.
• The encrypted archive: decrypts and tar -tf lists real entries.

On failure the drill emails me with the exact list of what broke; on success it logs and shuts up. First run: sixteen checks, all green — which, for the record, is the only moment I actually believed the afternoon's work.

The whole thing is ~150 lines of bash with one deliberate choice at the top: set -uo pipefail without -e, because a drill that exits on the first failed check tells you about one problem when you have three.

The second gotcha: --remove doesn't remove what --exclude hides

Small one, worth having in writing. While mirroring buckets I found a scatter of macOS ._* AppleDouble files — Finder-upload droppings — and added --exclude "._*" to the mirror so they'd stay out of the backup. Junk files already in the mirror stayed put: once a path is excluded, sync tools treat it as invisible on both sides, so --remove no longer considers the stale local copy extraneous. Excludes added after the fact need one manual find … -delete to clean up what already leaked through.

The checklist

If you self-host anything with state, the afternoon version:

1. Inventory the state. Every database in the cluster, every bucket, every volume. The gap is always the thing that got added after the backup script was written.
2. Prefer whole-cluster / whole-bucket tools (pg_dumpall, mc mirror over a bucket list) so new data is covered by default.
3. Small data → private git repo offsite. Rotation on disk, history in the repo.
4. Encrypt anything sensitive before it enters the repo — and re-evaluate "sensitive" whenever the backup's contents grow.
5. Change-detect before encrypting, or salted encryption will grow your repo on idle nights.
6. Get the key off the machine. A password manager counts. Two copies on the same VPS don't.
7. Write the drill. Freshness plus structural restore checks, on a schedule, that emails you only when something's wrong.

The drill runs next on the first of October. With any luck it'll have nothing to say — and for a backup system, a scheduled process that repeatedly proves it has nothing to say is the entire point.