I test restore my backup for the first time in 2 year. 2 of 5 things was broken

After greybeard_graham's tweet ("a backup you have never restored is a rumour") I feel personally attacked, so last weekend I finally do a real restore test. Not "I see the files in B2", a real restore to a new VM, start the apps, use them. Result: 3 OK, 2 broken. Nothing was lost because the originals are fine, but if the NAS died last month, I would lose all my scanned documents. Here is the story.

The setup

restic from the TrueNAS host to Backblaze B2, every night by cron, for 5 things: Immich (photos + DB), Vaultwarden, Paperless-ngx, Home Assistant, Gitea. ~1.4TB, about 8-9 USD a month on B2.

restic -r b2:hieu-backup:/tank backup /tank/apps /tank/photos \
  --exclude-file=/root/restic-excludes.txt
restic -r b2:hieu-backup:/tank forget \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune

Restore test

New Debian VM on pve3, install Docker, then:

restic -r b2:hieu-backup:/tank restore latest \
  --target /restore-test --include /tank/apps/vaultwarden

One by one for each app, then start it with the same compose file.

Broken #1: Immich database

The photos were there. But I was backing up the Postgres data directory while Postgres was running. Restore, start, Postgres complain and recover, then albums and face groups were half wrong. A file copy of a live database is not a backup, it is a random snapshot of files in the middle of being written.

Fix: dump first, then back up the dump.

docker exec -t immich_postgres pg_dumpall -c -U postgres \
  | gzip > /tank/backup/immich-db.sql.gz

Same for Vaultwarden, which is SQLite. That one was OK only because I already use the SQLite backup command:

sqlite3 /tank/apps/vaultwarden/db.sqlite3 ".backup '/tank/backup/vw.sqlite3'"

Broken #2: Paperless had NO documents

This one make me cold. Restore, start Paperless, log in, 2,300 documents in the list, click one: file not found. Click another: file not found.

My exclude file had this line, to skip the Jellyfin movies:

media/

And Paperless store all the documents in... /tank/apps/paperless/media/. For 2 years, my "backup" of Paperless was only the database. Every tax paper, every contract, the house papers, all scanned and the paper version shredded. Only on one NAS.

Fix: exclude with full path, never a bare name.

/tank/media/movies
/tank/media/tv

What I do now

• restic check --read-data-subset=5% every week. It download and verify a different 5% of the data each time
• Restore test every 3 months, in the calendar, to a different machine. Start the app, open 5 random items
• Every exclude is a full path
• Databases: dump first, always
• The restic password: it was in Vaultwarden. Which is inside the backup. Which need the restic password to restore. Now it is printed on paper, in an envelope, at my mother's house

Please, test your restore this month. Not "check the files are there". Restore and open them.