Retiring the fallback disks, and the watchdog that outlived them

For a while, the cluster had two old disks kept as fallback backup storage. Neither was in active use, but both were still wired into the system in ways that could surprise me later: a storage entry, a mount, and in one case a watchdog that was watching the mount.

I removed both on July 4 and July 5, 2026. Pulling the hardware was the simple part. The rest of it is what this post is about.

The two disks were different

One node had an internal SATA disk. The other had an external USB disk. The removal method followed from that.

The SATA disk needed the node shut down before it came out. I disabled the storage in Proxmox, backed up the fstab file, commented out the mount line, reloaded systemd, synced and unmounted the filesystem, and then shut the node down cleanly. The disk came out with the power off. The node booted back up afterward and I checked it.

The USB disk didn’t need a reboot, and I didn’t give it one. The same steps applied on the running node: disable the storage, back up fstab, comment the line, reload systemd, sync and unmount, confirm the mount was gone. Then I unplugged it.

What stayed, on purpose

Neither storage entry was deleted. Both stayed in Proxmox, disabled. The fstab lines stayed in place too, commented out. If either disk ever had to come back, undoing the removal would be a few edits and a plug, not a rebuild from notes.

I also didn’t prune any backups or move any VM disks. The old disks still hold old backup copies, and deciding what to do with those is separate work that isn’t done yet. The Proxmox Backup Server is still the backup target for both nodes.

The watchdog that outlived its disk

The USB disk had a watchdog. A systemd timer checked the mount on a schedule, a service ran the check, and a mount unit tied the whole thing to the disk’s path. Behind those sat a script. It was there to watch a disk I was about to remove.

Pulling the disk without retiring the watchdog would have left a check running against a mount that was never coming back. So I retired all of it: the timer, the service, the mount unit, and the script. I backed up the service and the script before removing them. Then I reloaded systemd, cleared the failed state, and checked that the old units were gone.

Booted is not the same as validated

The SATA node booted cleanly after its disk came out. I checked that the one boot drive was the only disk the system saw, that the expected guests were running, that the storage state was what I had left it as, and that no systemd units had failed. That told me the node came back. It did not, by itself, tell me the node was validated.

The USB node never rebooted, so its check was a running-state check. The distinction matters, and I wrote it down in the record. A clean boot is one data point about the moment the machine came up. A validated node is one where the checks have been run against the state it is actually in, and the closeout says which of the two you have.

The record for each removal says what was checked and when. I’d rather have that than a single line that says “done”, because the second one can’t be checked later.

What the sequence looked like

For either disk, the order was the same:

Disable the storage entry in Proxmox, but keep it.
Back up the fstab file.
Comment out the mount line, but keep it.
Reload systemd so it picks up the change.
Sync and unmount the filesystem, then confirm it is unmounted.
Shut the node down (SATA disk only), or leave it running (USB disk).
Remove the physical disk.
Check the node: boot disks, guests, failed units, and DNS where the node serves it.
Retire any automation that depended on the disk, and check that it’s gone.

Steps 1 to 3 are the reversible part. Step 7 is the only one that needs hands.

What’s left

Two things are still open. The old backup copies on those disks need a decision about what to keep. And the disabled storage entries and commented fstab lines can come out once I’m confident I won’t need them back, which is not yet.

The lesson I’d keep from this is small. Finishing a storage migration means retiring the automation around the hardware along with the hardware. And booting is only the first check.