The warning alerts that vanished after I added phone push

I added a phone push receiver to Alertmanager so alerts could reach me away from my desk. The new receiver worked. The configuration loaded. Nothing reported an error. Later, I found that warning alerts had stopped going to Discord.

That is an awkward failure mode for monitoring. The new path looked healthy, and the old path did not fail loudly. It simply stopped being selected. I had changed the route tree, not the warning rules or the Discord receiver. Alertmanager had done exactly what the new configuration told it to do, which was not what I meant.

The guesses that would have wasted time

My first thoughts were a bad Discord webhook, a missed reload, or a severity change in the rules. Any of those could explain missing warning messages. They were not good reasons to start changing credentials or rewriting alert rules, though. I needed to know whether Alertmanager would even select Discord for a warning before testing the delivery endpoint.

The live rule set had not been retiered. The critical rules were still critical and the warning rules were still warnings. The new phone path had been added as an Alertmanager child route. That put the most recent routing change ahead of the other guesses. It also gave me a way to test the problem without waiting for a real warning to fire.

The default was not a second destination

An Alertmanager route is a small tree. Every alert enters at the parent route. Alertmanager checks its children in order. Normally, the first matching child handles the alert and stops the search. A child with continue: true handles the alert and lets the search continue through later siblings. The parent receiver is the fallback only when no child matches at all. The routing documentation describes that last part, which I had missed.

Here is the shape of my mistake, with placeholder receiver names:

route:
  receiver: discord_warnings
  routes:
    - receiver: phone_push
      continue: true
    - receiver: discord_critical
      matchers:
        - severity="critical"

The first child has no matcher, so it matches every alert. Its continue: true does not mean “also use the parent default.” It means “keep looking at the next child.” A critical alert matches the second child and goes to the critical Discord destination as well as phone push. A warning does not match that second child. It has already matched the first one, so the parent fallback never runs. The warning goes to phone push only.

The YAML is valid. Both receivers can exist and pass a configuration check. A reload can return success. None of those checks asks whether a warning still reaches the destination I expect. That is why this broke silently.

One route test made it obvious

I ran amtool config routes test --config.file=alertmanager.yml severity=warning against the Alertmanager configuration. It returned only the phone push receiver. That one line was the proof I needed: the problem was route selection, before Discord delivery was even attempted. The test reads the file and sends no notification, so I did not need to create an outage or wait for an actual warning to see it.

I tested critical severity too. It selected phone push and the critical Discord receiver. This mattered because it showed the tree was not simply broken everywhere. The critical child was still doing its job. Only the warning path had lost its Discord destination.

There were no active alerts when I found the issue. That made the missing Discord messages less useful as evidence. The route test was deterministic: given a warning label, it showed exactly which receivers the current tree would choose. It did not depend on a rule firing at the right moment.

The final child was the fix

I kept the phone route and the critical route. I added one last child with no matcher, pointing to the normal Discord receiver:

route:
  receiver: discord_warnings
  routes:
    - receiver: phone_push
      continue: true
    - receiver: discord_critical
      matchers:
        - severity="critical"
    - receiver: discord_warnings

The fix is the final receiver line. A warning matches phone push, continues, skips the critical child, then matches the final child. A critical alert matches phone push and then the critical child. That child does not set continue: true, so the search stops before the final catch all. Critical alerts do not also reach the normal Discord channel.

I backed up the configuration, made the one line change, and ran the amtool configuration check. The check passed. I reloaded Alertmanager and confirmed the loaded configuration contained the new route. Then I repeated the route tests. Warnings selected phone push and normal Discord. Critical alerts selected phone push and critical Discord, with no normal Discord receiver added. The rule severities stayed unchanged.

Those were routing and reload checks, not a claim that I had watched a real warning arrive in Discord after the fix. A synthetic warning delivered end to end would test the webhook and notification path too. The route tests established the narrower fact I needed for this regression: the warning was no longer being discarded by the routing tree before it could reach Discord.

The rule I keep now

When I add a catch all child route, I write down what happens to each severity after that child matches. continue: true is a path to later siblings, not a promise to include the parent receiver. If a parent default used to handle ordinary alerts, I make that destination an explicit final child or I test why it is no longer needed.

I now pair a configuration check with route tests for at least a warning and a critical alert. A valid file and a successful reload answer whether Alertmanager can use the tree. The route tests answer whether the tree still sends each class of alert where I intended. In monitoring, both questions matter.