Patching four cloud servers took more than apt upgrade

I ran apt upgrade through Tailscale on a server whose pending updates included Tailscale. The package restarted tailscaled, and my SSH session disappeared in the middle of the upgrade. I managed this on both Oracle Cloud VMs. The connection I was relying on was part of the maintenance.

I started the round on September 29 and finished on October 4, 2026. Those two Ubuntu 24.04 VMs were half the job. I also had my netcup VPS, running Debian 13 with Caddy, Uptime Kuma, and ntfy in Docker, and a small GCP e2-micro running Debian 12 with 1 GB of RAM and no swap. By the end, the four servers had received 99 package updates and three kernel upgrades, with zero data loss and one VM that needed a console rescue.

The audit before the upgrades

I started with a read-only audit. Password and keyboard-interactive SSH login were off on all four, and automatic updates were enabled everywhere. Root login was off on three; the fourth still had Ubuntu’s default.

The pending package counts were 42, 35, 8, and 6. One Oracle VM had four security updates waiting. Three servers needed a reboot, and the Oracle VMs had been up for 15 weeks. SSH was open to the internet on all four, fail2ban was missing on three, rpcbind was listening on both Oracle VMs, and X11 forwarding was on everywhere.

My plan was one host at a time, least important first: update, reboot, validate, then move on. I deliberately kept public SSH open as a second way in. Closing it was a separate job, after testing console access.

Disconnecting myself was only the start

The upgrade command had a chained && reboot. That never ran after the Tailscale restart cut the connection, which turned out to be helpful. The package upgrade kept running in the background, then stuck at a keyboard-configuration debconf question. The terminal it wanted an answer from was gone.

On one Oracle VM, the new 7.0 kernel had been unpacked but its initramfs had not been built. A reboot at that point could have left it trying to boot a kernel with no initrd. I checked /boot before rebooting anything. The new kernel being present was not enough.

My first recovery attempt added another lost connection. I used pkill -f keyboard-configuration.config to stop the stuck configuration process. The -f option matches the full command line, including the SSH session running that command. I killed my own session.

On the second attempt I killed the stuck debconf frontend, dpkg, and apt by PID. I then ran dpkg --configure -a with DEBIAN_FRONTEND=noninteractive and --force-confold. Both machines finished with exit 0, no half-configured packages, and complete boot files for the newest kernel. Only then did I reboot them.

Both came back on 7.0.0-1011-oracle, with system state running, no failed units, and SSH working over both paths. Tailscale was at 1.102.4, the clocks were synced, and no package updates remained.

For the remaining hosts I used public SSH, set DEBIAN_FRONTEND=noninteractive, and made the reboot a separate step after checking the result. Running the upgrade inside tmux would also have kept it alive through a dropped session.

The VPS came back, including its monitoring

The netcup VPS upgraded nine packages, including Docker and Tailscale, without trouble. The containers restarted on their own. The kernel needed another look: apt upgrade had held back its metapackage because the update needed a new package. Rebooting then would have landed one kernel behind. I installed it explicitly, checked /boot, and rebooted onto 6.12.111.

All seven Uptime Kuma monitors reported UP within about 15 seconds of the restart. That was useful to see after it came back, but while the box was down, nothing could alert: the alerting (Kuma and ntfy) lived on the same box. That gap was still there after the successful patching.

The small VM had already stopped updating

Every name lookup failed on the GCP VM. Tailscale’s accept-dns setting had replaced /etc/resolv.conf with its resolver, which forwarded to my home Pi-hole. This VM had no route to my home LAN. It was the same root cause as the standby router that couldn’t tell time, this time showing up during package maintenance.

Automatic updates had been enabled but failing quietly. The pending list included security updates for openssl, libssl3, xz-utils, and liblzma5. I ran tailscale set --accept-dns=false and restored /etc/resolv.conf as a symlink to systemd-resolved’s file. Tailscale had left a backup showing the original setup.

Once DNS worked, the next upgrade froze the VM while setting up the Google Cloud CLI package. It was an 88 MB package with a slow, memory-hungry setup on a machine with 1 GB of RAM and no swap. Neither SSH path answered.

After a stop/start, it came back with a new ephemeral public IP. I finished from the browser SSH console: removed google-cloud-cli, which this monitoring VM did not need, completed dpkg, and restarted the guest agent. The pending count reached zero, DNS worked, and automatic updates were working again.

Patching finished; hardening did not

Public SSH is still open. Before closing it behind Tailscale, I need to test console break-glass access on each provider. The small VM still needs swap. The fail2ban, rpcbind, and X11 cleanup belongs in the hardening playbook, along with a monitoring vantage point that is not on the same box as the alerting. That is the next post’s work, not something these package updates quietly solved.