entry-48
One Caddy VM and one wildcard certificate
I put the lab services behind one internal Caddy on the proxy VM, with a single wildcard from DNS-01 and a Cloudflare token locked to the home address. The lab names were too long to use, the Pi-holes blocked the new client, and Chrome flagged n8n.chrisalorenzo.com/signin, so n8n became flow.
revisions rev. 1 first set 2026-10-23 last revised 2026-10-23 8 min read
Update (2026-10-19): The Forgejo proxy follow-up records the HTTPS move that was still pending in this account.
I wanted the lab services on names I would actually type, with a
certificate a browser would accept, and with those names kept off
the public internet. The design I approved on 2026-10-12 is small.
One Caddy VM. One wildcard from Let’s Encrypt over DNS-01.
Pi-hole records on the
LAN. Tailscale split DNS so the same names work away from home,
through the subnet router,
with no port forwards. The proxy stays off the tailnet. Off-LAN
clients reach it at its LAN address. UniFi stays at
https://192.0.2.1.
The VM and the first certificate
The proxy VM is VMID 296 on the second Proxmox node, at 192.0.2.84. That address
used to belong to the first subnet-router VM. On 2026-10-12 nothing answered there, and
it sits outside the UDM DHCP pool (.100 to .199). The VM is 1
vCPU, 1 GiB, and 8 GiB, cloned from template 9002 with OpenTofu
from the private infrastructure repo.
The first certificate was *.lab.chrisalorenzo.com. I did not want
the Caddy token editing the public zone, so the plan was a permanent
DNS-only CNAME, _acme-challenge.lab.chrisalorenzo.com, pointing at
lab.acme.stayz3ro.dev. That zone is already redirect-only. Caddy
writes the TXT there with dns_challenge_override_domain, because
certmagic does not follow the CNAME when it writes. The token is
Zone DNS Edit and Zone Read on stayz3ro.dev only. Pins were Caddy
v2.11.7 and caddy-dns/cloudflare v0.2.4. If delegation failed, the
fallback was a full-zone token on chrisalorenzo.com with
Cloudflare’s client IP filter locked to the home address.
On 2026-10-13 I created Proxmox user tofu-phase3@pam and a token
that can clone template 9002 and configure VM 296.
VM.GuestAgent.Audit on that role cleared an earlier 403 on VMs 105
and 106. The plan was one add. Apply finished in 2 minutes 49
seconds, then waited. The stock Debian image has no
qemu-guest-agent, and OpenTofu waits for that package even when
the template’s agent option is already on. I installed it by hand.
The guest got a full upgrade, Docker 29.9.0 with the compose plugin,
and UFW default deny: 22 and 443 from 192.0.2.0/24 only. No
Tailscale. Cloud-init had already set the resolver to the Pi-hole
VIP, 192.0.2.20.
The Cloudflare token is encrypted with systemd-creds on the host
key. There is no TPM. The unit loads it with
LoadCredentialEncrypted=. On systemd 252 (Debian 12), plain
LoadCredential= passes the blob through undecrypted. I checked by
length only and did not print the value.
Caddy, built on the VM with the Cloudflare module, got
*.lab.chrisalorenzo.com from Let’s Encrypt on the first try, in
about 9 seconds, valid to 2027-01-11. The ACME TXT was cleaned up.
DNS did not exist yet, so I used curl --resolve at .84. The three
Proxmox nodes, PBS, Portainer, Alertmanager, and n8n returned 200.
Grafana and Prometheus returned 302. git returned 502 on purpose.
Forgejo stays on its Tailscale Serve URL while the pull
mirrors on the relay VPS still use it.
The Pi-holes returned 502, dial tcp ...:443: i/o timeout. Each UFW
allowed 443 only from the monitoring VM (192.0.2.94, comment
nebula-sync). The proxy is a new client. The path and the service
were fine. The design never listed this rule. I added one narrow
allow on each node, TCP 443 from 192.0.2.84, not 443 to the
subnet. Both then returned 302 to the login page.
Four smaller mismatches went into pull request 4 in the private infrastructure repo and the
runbooks. Docker Hub’s Caddy tags are 2.11.7 with no leading v,
and xcaddy wants v2.11.7. docker compose --env-file only
interpolates the compose file, so CLOUDFLARE_API_TOKEN has to be
declared under environment:. Debian 12 has no /usr/local/libexec.
The cluster CA (mode 640, root:www-data) and the Pi-hole CA bundle
on the monitoring VM (directory mode 700) are public certificates, and the
admin user cannot read them. I staged each to /tmp with sudo and
copied with scp -3. A direct copy makes the source host try to SSH
to the proxy VM.
The names were too long
grafana.lab.chrisalorenzo.com looked fine on paper. It is too long
to type. I only noticed after the certificate existed and I tried to
use it. On 2026-10-14 I dropped the lab label.
The names are flat: graf, prom, alerts, docker, pbs, hypervisor-a
to hypervisor-c, dns1, dns2, and git later. docker is the function.
Portainer answers it today, so a later move to Arcane keeps the URL.
auth is a placeholder. Public names in the same zone stay public:
www, blog, status, and a. Renaming the VMs is separate.
SSH aliases, Tailscale machine names, Prometheus labels, Nebula Sync,
and the Pi-hole certificate names under .home.arpa follow the
hostnames.
The replacement certificate is *.chrisalorenzo.com. Its challenge
name, _acme-challenge.chrisalorenzo.com, is the same name Cloudflare
uses for the public site’s edge certificate. I did not put the CNAME
delegation there. I had not ruled out breaking that renewal. I used
the fallback instead: Zone DNS Edit and Zone Read on
chrisalorenzo.com, with Cloudflare client IP filtering locked to
the home IPv4. The proxy VM has no public IPv6. If that address changes,
renewal fails until the filter is updated. Caddy renews 30 days
early. Let’s Encrypt issued the new wildcard on the first try,
valid to 2027-01-12. The expiry alert for an address change is still
unwritten, and the old _acme-challenge.lab CNAME plus the
stayz3ro.dev token are not cleaned up yet.
While switching tokens I checked the new one with curl, and the value
was on the terminal. I rolled it once and set the credential file to
mode 0600. The runbook reads the token with systemd-ask-password,
which keeps it out of shell history and off disk in plaintext. The
first successful issuance is the real check.
At home and away
On 2026-10-12, dns.hosts was empty on both Pi-holes. UDM DHCP
already gives every LAN client 192.0.2.20 as its only resolver,
so the LAN does not need Tailscale for this.
On 2026-10-14 I set 11 records on the primary DNS node, each one
192.0.2.84: the short names above, with n8n in the set that
day and git still left for later. The runbook’s single-quoted form
was wrong. FTL 6.6.2 wants a JSON array of double-quoted
"<ip> <name>" strings. Single quotes fail with not valid JSON.
The write needs sudo (ssh -tt) and it replaces the whole array, so
a partial paste drops the rows it omits. All 11 resolve to .84
through the VIP. No public record exists for any of them.
graf.chrisalorenzo.com opens with a valid certificate. The secondary DNS node
receives the list at the next Nebula Sync. Sync copies dns.hosts
and not a dnsmasq wildcard, which would sit only on the primary DNS node and
drift. A typo should be NXDOMAIN, not a silent hit on the proxy.
Off the LAN, split DNS is one restricted entry per name, all of them
to 192.0.2.20, not the whole chrisalorenzo.com zone. On
2026-10-12 the tailnet had MagicDNS for ts.net only, no global
resolver, and no split entry for this zone, so these entries do
real work. The design’s caution, that 100.100.100.100 already
forwarded to the VIP, did not match that config. Sending the bare
zone through
the home Pi-hole would also send www, blog, status, and a,
and those would stop resolving away from home when the home network
is down. tailscale dns status on the workstation lists 11. Queries
go to the VIP over the subnet route 192.0.2.0/24. Same
dependency as off-LAN Forgejo SSH: if both subnet routers are down,
off-LAN DNS and the proxy are both unreachable, and the LAN is fine.
Grafana, on the monitoring VM, now has
GF_SERVER_ROOT_URL=https://graf.chrisalorenzo.com/. I backed up the
compose file, recreated Grafana, and checked the redirect and
/api/health. On the n8n VM (105) the env file uses the proxy name,
with N8N_PROTOCOL=https, N8N_SECURE_COOKIE=true,
N8N_PROXY_HOPS=1, and N8N_WEBHOOK_URL beside the deprecated
WEBHOOK_URL. Backups are .bak-proxy-20261014. Plain
http://192.0.2.82:5678 login no longer works, by design.
Alertmanager’s direct webhook is unchanged. Narrowing backend ports
to the proxy only waits until this has been stable for a week or two.
Chrome renamed n8n to flow
Chrome, Standard protection, normal window, showed a Safe Browsing
“Dangerous site” page for n8n.chrisalorenzo.com/signin. Google’s
transparency report had no data for the name. Incognito, local list
only, was clean, so the verdict was the real-time check. It treats
an n8n.<unknown>/signin URL as a brand phishing page.
SafeBrowsingAllowlistDomains is ignored on this unmanaged Windows
PC. Chrome says “This policy is blocked.”
flow.chrisalorenzo.com on the same backend loaded clean. I moved
the Caddy matcher, the Pi-hole record, the split DNS entry, and the
n8n URL settings, and retired the n8n name. I filed a
false-positive report for the old one. Internal names should avoid
product brand names that have a hosted login. n8n hit that.
grafana has been fine so far. The name in DNS is graf.
What I would do differently
I would type the URL before building DNS, certificates, and service
config around it. I would put each backend firewall change in the
design, one narrow rule per new source. I would bake
qemu-guest-agent into the image so apply does not wait. I would
not prove a DNS token with curl. A hidden prompt, then the first
successful issuance, is the check. A value that lands on a terminal
gets rolled. I would add the certificate-expiry alert before
depending on a renewal that fails when the home address changes.
Still open: the hotspot test, deleting the lab ACME CNAME, revoking
the stayz3ro.dev token, adding VMID 296 to phase26-core-infra-pbs
and running one backup, Forgejo on git after the mirrors move, and
the secondary DNS node picking up the host list on the next sync.