A well-written alerting rule (see the article on symptom vs cause alerts) decides when to fire an alert, but decides nothing about what happens next. Without explicit Alertmanager configuration, a single incident affecting 50 pods simultaneously generates 50 identical notifications, a storm that drowns the useful signal under its own volume.
Routing: directing each alert to the right team
Alertmanager routes every received alert through a rule tree (route), deciding the destination (Slack, PagerDuty, email) based on the alert’s labels, not its text content.
route:
receiver: default
routes:
- match:
team: platform
receiver: platform-pagerduty
- match:
team: billing
receiver: billing-slack
Without an explicit routing tree, every alert lands on the same channel, regardless of which team is actually concerned: a useful signal for one team becomes noise for every other one receiving the same notification with nothing actionable to do about it.
Grouping: one notification, not fifty
Grouping (group_by) bundles several alerts sharing the same values across a set of labels into a single notification, instead of sending one per individual alert.
route:
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
Fifty pods failing their readiness probe at the same time, all tied to the same alertname and the same cluster, produce a single grouped notification listing all fifty affected pods, instead of fifty separate alerts that saturate the notification channel and make the real signal (“something is affecting the whole cluster”) less visible than the noise it generates.
group_wait sets the delay before a new group’s first notification (time to gather other similar alerts before sending); group_interval sets the minimum delay between two notifications for an already-notified group.
Inhibition: suppressing the alert that says nothing new
Inhibition suppresses an alert when another, judged more significant, is already active, avoiding notifying an expected consequence of an already-reported cause.
inhibit_rules:
- source_match:
alertname: NodeDown
target_match:
alertname: PodNotReady
equal: ['node']
This rule suppresses any PodNotReady alert on a node already reported as NodeDown: pods on that node obviously aren’t responding anymore, a direct and expected consequence, not additional information deserving its own notification. Without inhibition, a node going down generates an alert for the node itself, plus one alert per pod that was running on it, a noise multiplier for a single real incident.
Silences: cutting off temporarily, without touching configuration
A silence temporarily disables notifications for alerts matching a precise pattern, without modifying routing configuration or the rules themselves, typically during a planned maintenance window.
# Silences every alert from the "staging" cluster
# for 2 hours, without touching configuration
amtool silence add cluster=staging --duration=2h
A silence expires automatically at the end of its duration, unlike a manual rule change that sometimes gets forgotten and never restored after maintenance.
Takeaway
A well-written alerting rule isn’t enough on its own: without explicit routing, every alert lands in the same place; without grouping, an incident touching several targets at once generates one notification per target instead of a single grouped one; without inhibition, an expected consequence of an already-reported cause generates its own noise. Silences handle maintenance windows without touching permanent configuration. These four mechanisms, combined, turn a technically functional alerting system into one genuinely usable on-call, one of the pillars of reliability and observability that never drowns the signal under its own volume.