Building a backup system I actually tested

A backup you have never restored is a theory. I had scheduled vzdump jobs running since the Proxmox move, and they produced archives. But a backup job finishing and a backup being usable are different things, and I had never proved the second one.

This is the arc that closed that gap: stand up Proxmox Backup Server, prove it with real restore tests, then build and prove a separate encrypted offsite layer. It took a while and it is worth writing down because most backup writing stops at the schedule.

Proxmox Backup Server first

The lab got a third Proxmox node, pve03, and on it a VM running Proxmox Backup Server with a dedicated passthrough disk. PBS became the backup-svr-pbs storage on all three nodes, with the pbs-main datastore. Scheduled backups ran against it, with prune, verify, and garbage collection baselines configured.

PBS also got its own monitoring. The UI was probed through Blackbox, and the datastore’s disk usage was exported through Node Exporter into the existing Grafana dashboards. A backup server that fills its disk quietly is not much of a backup server.

Verify is not restore

The scheduled jobs and verify runs looked healthy. Verify checks the integrity of stored chunks. It does not prove you can recover a VM from them. I treated the two as separate, because they are.

The proof is blunt. Restore a real backup to a throwaway VMID and boot it. I restored a non-critical production VM, the self-hosted RustDesk server, to a temporary VMID on pve03, with its network interface removed before first boot so it could not conflict with the live VM. It booted. Then I destroyed the temporary VM and confirmed it was gone.

That is the whole test. Restore to a new identity, boot with networking removed, confirm, destroy. A backup that cannot clear that bar is a backup I should not be depending on.

The offsite layer

A local PBS is not disaster recovery. It protects against a failed VM or a bad update, not against the whole lab going down at once. So the next step was an independent offsite copy.

The first candidate, a second PBS instance offsite, got rejected after a provenance audit found it had never actually been approved as an architecture. That was embarrassing and useful. The plan had drifted from an agent recommendation into implementation without a real operator decision, and catching it meant re-examining what I actually wanted.

What I wanted was smaller than a second PBS. Two layers:

• PBS stays as the Proxmox-native VM recovery layer.
• A separate encrypted file-level engine for the data that should not depend solely on PBS.

For that second layer I evaluated restic and Kopia with the same test: back up a synthetic dataset, check it, delete the source, restore it, and compare SHA-256 hashes. Both passed. I chose Kopia, mostly on encrypted deduplicated snapshots and a policy model that matched how I would actually run it.

The B2 proof

The real test was proving that encrypted offsite backup works end to end. A disposable Backblaze B2 bucket, a scoped application key, a Kopia repository pointed at it. Snapshot a ten megabyte synthetic dataset, run kopia repository validate-provider, restore into a disposable directory, and compare SHA-256 hashes. They matched.

Then teardown. The bucket and key were deleted, and the local Kopia config and restore directory were removed. No real data ever touched it. A proof of concept that leaves state behind is not disposable.

One flagged deviation along the way: a status command echoed the B2 application key ID, not the secret, into a transcript. The key was revoked at teardown anyway. It is a good reminder that even the readout of a tool can leak something, and that disposable credentials exist to be revoked.

What this actually bought

Two things that did not exist before. A Proxmox-native restore path that has been proven by booting a restored VM. And a separate encrypted offsite layer whose round trip, out to B2 and back with matching hashes, has been proven. Neither was proven by the scheduled jobs alone.

The offsite POC is a proof, not production. A permanent bucket, a real dataset selection, and scheduling and monitoring for Kopia are still ahead. The point is that the design is no longer a theory either. When I do turn it on for real, the scary parts, the encryption, the provider, the restore path, have already been exercised.

Backups are one of the few places where the honest version of the work is a test you run, not a config you write. Restore something you need. Then tear it down and do it again.