Putting Forgejo behind the proxy without opening it

I moved Forgejo onto the internal reverse proxy. It was the last service in the queue. The proxy already fronts the monitoring stack, the Proxmox nodes, the Pi-holes, and n8n, each one a Caddy site behind a single wildcard certificate, each on a short flat name under chrisalorenzo.com. Forgejo was deliberately left for last because it is the one service whose own configuration had to change, not just its DNS.

The setup I was starting from

Forgejo runs in Docker on its own VM at 192.0.2.83. Its HTTP listener was bound to 127.0.0.1:3001, loopback only. That was on purpose. An earlier hardening pass had taken port 3000 off the LAN because logins crossed the LAN in clear text. The only way in was a Tailscale Serve URL that forwarded to the loopback port. Git over SSH was separate, published on 192.0.2.83:2222.

The goal was to publish a second binding, 192.0.2.83:3000, that exactly one host could reach: the proxy at 192.0.2.84. Caddy would reverse-proxy git.chrisalorenzo.com to it. I also needed to set the two fields Forgejo uses to build URLs, ROOT_URL and SSH_DOMAIN, and leave SSH on port 2222.

ufw does not see Docker published ports

I expected to add a ufw rule and be done. That is not how it works. When Docker publishes a port, the packet is DNAT’d and travels through the FORWARD chain, not the INPUT chain where ufw’s rules live. A ufw allow on 3000 would never be consulted, and ufw status would still show no rule for 3000 after the change. The control for a Docker published port is the DOCKER-USER chain, which Docker creates for exactly this.

That chain runs after Docker has already done the DNAT. By the time a packet reaches it, the destination has been rewritten from the published address and port to the container’s address and port. A --dport match on the published port does not fire; the match that works is --ctorigdstport, the original destination port before the rewrite. The port 2222 mapping is the clear case: it maps to container port 22, so a rule that matches --dport 2222 would never see it.

A rule for one container has to name that container

The existing rules for SSH on 2222 had the port match but no destination match. They allowed the LAN range and the tailnet range to 2222 and dropped everything else on that port. The drop was the problem. With no destination match, it evaluated every container’s forwarded traffic on the original port, including outbound connections from containers that had nothing to do with SSH. This is the same failure the lab wrote down after an earlier incident, where an unscoped drop quietly broke containers’ new outbound connections while old pooled connections kept working, so nothing looked wrong for days. A restriction meant for one container needs a destination IP match on both its allow rules and its final drop. Scoping only the allows leaves the broad drop in place.

The address to match is the container’s post-DNAT address, 198.51.100.2 on the forgejo_default network, not the VM’s LAN address. That address changes when the container is recreated, so the script reads it fresh every time, refuses to accept anything in 192.0.2.0/24, and rewrites the destination on every rule. After any docker compose up on that VM, the address is re-read and the rules updated before anything else.

Drop the port before you publish it

The order mattered. I put a drop for port 3000 into the chain first, above the final RETURN, before the port was published at all. That way there was never a window where the new port was open. The chain ends in a RETURN that allows every other forwarded packet, so the 3000 drop has to sit above it. The hold was a single unscoped drop; the next step replaced it with the scoped pair.

The change, in one edit and one reload

The publish and the scoping went in as a short sequence. First, back up the compose file, the env file, and /etc/ufw/after.rules. Then install the hold drop and reload. Then add the LAN binding beside the loopback publish and set FORGEJO_PRIVATE_URL to the new https://git.chrisalorenzo.com/ value, and recreate the container. Then one script reads the container address and writes all five rules as a single block, followed by one ufw reload. The port 2222 allows and drop got the destination match in the same write, so the whole change has one rollback: restore after.rules and the two files, reload, recreate.

The block lives in a second *filter section in /etc/ufw/after.rules, with the chain declared as :DOCKER-USER - [0:0] so a ufw reload replaces it cleanly. I did not use iptables-persistent; it removes ufw on Debian.

What the validation actually proved

Validation was the interesting part, because the failure I was guarding against does not show up in the obvious test. From the proxy, a curl to the new port returned 200, and the same request through Caddy with --resolve returned 200. From another LAN host, the same request returned 000, which is what curl reports when a drop times out. That is the drop working.

The test I cared about most was a fresh outbound connection from the container. An unscoped drop would break exactly that, and an old established connection would not have told me, because the chain’s first rule lets established traffic through. So I opened a TCP connection to github.com:443 from inside the container’s network namespace. It printed github_443_open. I also confirmed SSH on 2222 still worked from another host and that the Tailscale Serve URL still answered. Serve was a stop condition: if it had broken, I would have rolled back even though the new path worked.

The counters on the 3000 rules both moved after the test requests, which confirmed the new rules were the ones handling the traffic.

The names, and keeping the old path alive

Then the names. The Pi-hole keeps its local records in one list, and setting the list replaces the whole thing, so the change was one command that re-sent all eleven existing names plus two new rows: git pointed at the proxy, and gitssh pointed at the Forgejo VM itself. Git over SSH is not proxied; Caddy speaks HTTP only. The second Pi-hole picks the list up on the next sync, and the Tailscale split DNS entries for both names were already in place, so off-LAN devices resolve the same names through the Pi-hole VIP. I logged in once at https://git.chrisalorenzo.com/ in a browser, and the clone URLs on a repository page now show the new names.

I kept the old path running. Tailscale Serve still forwards the Serve URL to 127.0.0.1:3001, and it stays that way for a week. The twenty-one pull mirrors on the off-LAN host still fetch from the Serve URL. Moving them is a separate step, and I will only move them while Serve is still on so a failure can fall back. After the mirrors have run a clean cycle, Serve comes off and the loopback publish goes away. None of that has to happen on the same day as this change.

What I would do differently

A few things. The destination match on the port 2222 rules should have been there the day those rules were written. The pattern was already known here, and leaving the drop unscoped was a latent bug that only the port 3000 work forced me to fix. I would write the drop-first ordering into the checklist for every future published port instead of treating it as a detail of this one service. And I would keep a “is this rule scoped to a container address” check as a standing review item, because the address changes on recreate and a stale match fails quietly: inbound still looks right while outbound is what actually breaks.

The service is behind the proxy now, on a name short enough to type, with exactly one host allowed to reach the cleartext port behind it. The old path is still there in case I need it, which is the point of doing this in stages.