Monitoring the DNS stack · Chris Lorenzo

After the HA DNS build, the setup worked but I could only tell it worked by checking it manually. This phase added monitoring, which is what turned it from something I configured into something I operate.

The stack is the standard one: Prometheus for metrics, Grafana for dashboards, Blackbox Exporter for probing endpoints, Node Exporter for host metrics, and Alertmanager routing alerts to Discord.

Standing it up

Grafana on a fresh install, before any dashboards or data sources.

Grafana "Welcome to Grafana" page on localhost:3000, the
getting-started panel with no dashboards yet

Node Exporter on both Pi-hole nodes gives host-level metrics: disk, memory, CPU, uptime, and a simple up/down signal.

Grafana "Node Health" dashboard: root disk free, memory used, and CPU
for both nodes, with uptime and a node-up count

The more useful probes are the Blackbox ones that check DNS itself: resolution against each node directly, and against the Keepalived VIP that clients actually use. A green host with a wedged resolver still shows as a DNS failure that way.

Testing the alerts

A dashboard shows state. An alert is a claim that something will happen when state goes bad, and that claim is worth nothing until you’ve seen it fire.

So I broke a node on purpose. Stopping Node Exporter on the backup Pi produces a SingleNodeDown alert: pending first, then firing, then a Discord message.

Prometheus graph of ALERTS for SingleNodeDown: the alert moving through
pending and firing states for the backup DNS node

Bringing the node back clears it, and Alertmanager sends the recovery notification.

Prometheus graph showing the SingleNodeDown alert resolved after the
node came back

I did the same for the failover case: force the VIP to move, confirm the per-node and VIP probes track it, confirm the dashboards reflect it, and confirm the alert state is what I expected throughout.

What monitoring changed

Before this phase, “is DNS healthy” was a question I answered by SSHing in and running commands. After it:

the state is visible at a glance
a real failure produces a notification I’ll actually see
I’ve watched the alerts fire and recover, so I trust them
the failover path is monitored end to end, not just the two boxes

That’s the difference between a configured service and an operated one. The next phase made the whole thing reachable from outside the house without opening any ports.