One firewall bug, two hosts, and a weekly report

Tonight was a long one. Two new VMs on the Proxmox cluster, one for n8n and one for Forgejo, both built with OpenTofu, hardened, backed up, and restore-tested. Along the way I found a firewall bug on those VMs that had a twin on my public VPS, where it had been quietly lying to my status page for almost a week. Then I built the first real n8n workflow on the new VM, and it found four bugs of its own before it sent a single correct message.

The build, briefly

Both VMs are full clones of my Debian 12 cloud-init template, created with a tightly scoped Proxmox API token. That token was its own small adventure: the user landed on a different realm than my plan assumed, every privilege had to be granted twice (once to the user, once to the privilege-separated token), and three privileges I had not planned for only showed up as 403s during the apply. One of those 403s left a correctly cloned VM marked tainted, and tofu untaint saved it from an unnecessary destroy and rebuild.

After that came the usual baseline. Docker and Tailscale from their own signed repositories, UFW with default deny inbound, and both VMs added to the nightly Proxmox Backup Server job. Forgejo got a tailnet-only HTTPS certificate through Tailscale Serve. Then a fresh backup, and a restore of that backup to a temporary VM with its network card removed. I checked through the guest agent that the Forgejo database and the test repository were really in the restored copy, and then deleted the temporary VM.

One note on that restore test. My first attempt used the backup from before Forgejo was installed, so of course the data was missing. That was not a failed restore. It showed that a restore test has to use a backup taken after the data you care about actually exists.

Docker publishes ports around UFW

This one was not in the plan at all. It came up while I was reviewing the maintenance runbook after the build.

When Docker publishes a container port, it adds a DNAT rule that rewrites the destination to the container’s bridge address. From that point on, the packet is forwarded traffic, not traffic addressed to the host. UFW’s normal rules live on the INPUT chain, so they never see it. Docker gives you a hook for this, the DOCKER-USER chain, which runs before Docker’s own forwarding rules. On both VMs, it was empty.

The practical exposure was small. There are no port forwards on my router, and the source ranges I meant to allow cover almost everything on my flat LAN anyway. But “UFW allows only these ranges” was simply not true for the published ports, and that would matter the day I segment the network.

My first fix made things worse. I installed iptables-persistent to save the new DOCKER-USER rules across reboots, and on Debian that package conflicts with ufw. It removed ufw without complaint, which also reset the n8n VM’s INPUT policy to accept. I caught it before touching the Forgejo VM, reinstalled ufw, reapplied the whole baseline, and checked it again.

The fix that stuck goes through UFW itself. I appended a second *filter block to /etc/ufw/after.rules that declares the DOCKER-USER chain and adds the allow and drop rules. Declaring the chain is what makes ufw reload replace its contents instead of stacking duplicates. Two details mattered here:

Match on --ctorigdstport, not --dport. By the time a packet reaches DOCKER-USER, Docker’s DNAT has already rewritten its destination, so a plain --dport match never fires and never tells you.
Read the source, not blog posts. The posts I found disagreed with each other. Docker’s own source and UFW’s own code agreed with each other.

Simplified, with the LAN range left out, the n8n version looks like this:

*filter
:DOCKER-USER - [0:0]
-A DOCKER-USER -p tcp -m conntrack --ctorigdstport 5678 -s <lan-range> -j RETURN
-A DOCKER-USER -p tcp -m conntrack --ctorigdstport 5678 -s 100.64.0.0/10 -j RETURN
-A DOCKER-USER -p tcp -m conntrack --ctorigdstport 5678 -j DROP
COMMIT

Notice that every rule names a port. Keep that in mind for the next section.

The same bug, on the VPS, pointing the other way

Around the same time, my public status page started showing the blog, the portfolio, and the HTTPS edge as down. All three were fine. I checked them from outside and they answered normally.

The cause turned out to be a firewall change I had made on the VPS almost a week earlier. To stop people from bypassing Cloudflare and hitting Caddy directly by IP, I had added DOCKER-USER rules that allowed ports 80 and 443 only from Cloudflare’s published ranges and dropped everything else. Those rules matched on source address and port. They did not match on destination.

DOCKER-USER sees all forwarded Docker traffic, not just traffic coming in to Caddy. That includes every container’s own outbound connections. When the Uptime Kuma container opened a new connection to some site on port 443, its source address was not a Cloudflare address, so it fell through to the final DROP.

So why did it take a week to show up? Connections that were already established were still allowed, and Kuma’s HTTP client keeps its connections open for a long time. Everything kept working on those old connections. The first time Kuma needed a genuinely new connection, it hit the same rule that was meant to stop strangers, and the status page started reporting real sites as down.

I ruled things out in order, each checked directly: Docker’s NAT rule for that network was present and passing traffic, the default route was the normal public interface and not Tailscale, and the conntrack table was nearly empty. The test that settled it was a raw-IP HTTPS request to 1.1.1.1 from inside the Kuma container. With DNS out of the picture, it still timed out. Then DOCKER-USER showed a catch-all DROP with about 405 thousand packets on its counter.

The fix was to add a destination match. The same Cloudflare ranges, now scoped to Caddy’s container address on both the allow rules and the final drop. Everything else forwarded through Docker, including every container’s outbound traffic, no longer touches that policy. I saved the full ruleset with iptables-save before changing anything. Afterward, the raw-IP request from Kuma got a 301, a direct request to the VPS from a non-Cloudflare address still timed out, and the status page went green.

Same chain, same missing field, two different symptoms. On the lab VMs, there was no rule at all, so inbound traffic to published ports went around UFW. On the VPS, a rule without a destination match blocked outbound traffic it was never meant to touch. The rule I am keeping: any DOCKER-USER rule meant to protect one container needs to name that container, by destination address or by original destination port. Otherwise it applies to every container on the host.

W11: a weekly report that has to earn its trust

With n8n up, I built the first workflow meant to run there for real. W11 sends one ntfy push every Sunday that answers “is the lab healthy?” It pulls from four sources:

Uptime Kuma’s public status page: what is up, what is down, average 24 hour uptime.
Alertmanager: what is firing, grouped by severity.
A TLS handshake against each public host, to count days until its certificate expires.
Proxmox Backup Server’s task list for the last seven days, counted by task type.

The priority goes up to high on warnings and to urgent on anything down, failed, or close to expiring. The rule I cared most about: a source the workflow cannot read shows up as UNREACHABLE at high priority. It never shows up as an all clear. The status page incident above is exactly why. A health report that stays quiet when it cannot see is worse than no report.

The workflow JSON has no secrets in it. The PBS token and the ntfy token live in n8n Header Auth credentials that I create in the UI on import.

Getting the first correct push took four fixes:

The TLS module was not allowed. The certificate check is a Code node that needs Node’s tls module, and n8n’s Code sandbox blocks built-in modules unless you allowlist them. I had the wrong environment variable name. The one that applies to the task runner is N8N_RUNNERS_ALLOWED_BUILT_IN_MODULES=tls.
Both credential header names were wrong. Both Header Auth credentials need the header name Authorization, with the scheme in the value (PBSAPIToken=... for PBS, Bearer ... for ntfy). Both were misconfigured, and each one failed in its own way.
The PBS token could not see what it needed. It was the same lesson as the Proxmox token from the build, on a different product: the token’s permissions were scoped too narrowly for the task list, and the fix was the narrowest grant that let it read.
A worker type string was wrong. This one was in the report logic itself. The backup section finds verify tasks by worker type, and the string it matched on was not the one PBS uses. PBS calls them verificationjob. With the wrong string, the verify check could never match, no matter how many verify jobs succeeded. Comparing against the real task list is what caught it.

The README has a “first live run must prove” list for exactly this reason: Kuma counts match the status page, alerts match the Alertmanager UI, certificate days match openssl s_client, PBS counts match the task log, and stopping one source turns its section into UNREACHABLE. A workflow that imports cleanly has proven nothing yet.

What carried over

The same pattern showed up three times tonight. A rule that looked complete was missing one field, and the system did exactly what the rule said, not what I meant. On the VMs, it was a firewall chain that UFW never consulted. On the VPS, it was a drop rule with no destination. In W11, it was a worker type string that was close but wrong. In each case the fix came from looking at what the system actually saw: the packet counter, the raw-IP request from inside the container, the real task list.