It is 3am and the pager fires: HighCPUUsage, node at 92%. You open a laptop, look around, and everything works. Users are asleep, the batch job that spiked CPU finished on its own, and you go back to bed annoyed. Two weeks of that and people start muting the channel. The alert was technically correct and completely useless, because it told you about a cause that may or may not have a symptom.

The single most useful idea in alerting is this: page a human on symptoms, and let causes stay on dashboards. High CPU is not a problem. Slow responses are. If the CPU is at 92% and the service still meets its latency target, there is nothing to do. Alert on the thing your users actually feel, and you get a pager that means “something is broken right now” instead of “a number crossed a line”.

Symptom-based versus cause-based

A cause-based rule watches an internal resource: CPU, memory, queue depth, disk usage. A symptom-based rule watches the contract with the user: error rate, latency, availability. The problem with cause-based alerting is that the mapping from cause to symptom is not one-to-one. High CPU sometimes hurts and usually does not. A full disk is a genuine emergency on a database node and a shrug on a stateless replica. Every cause needs a human to decide whether it matters, at 3am, which is exactly the decision the alert was supposed to make for you.

This does not mean deleting your resource metrics. Keep them; they are how you diagnose an incident once a symptom alert has woken you. The rule of thumb: symptoms page, causes inform. Disk-filling and certificate-expiry are the honest exceptions, because they are causes with a guaranteed, predictable symptom in the near future, so alerting early is the whole point.

How for and severity actually work

Prometheus evaluates alerting rules every evaluation_interval (1 minute by default). When a rule’s expression returns samples, the matching alerts enter the pending state. They only become firing, and get sent to Alertmanager, once the expression has stayed true continuously for the duration in the for field. A single evaluation where the expression returns nothing resets the timer.

That for clause is your first and cheapest noise filter. An expression with no for fires on a single scrape, so one slow scrape or one transient spike pages you. for: 10m says “I only care if this is still true ten minutes from now”, which filters out exactly the self-healing blips that generate 3am false alarms. The trade-off is detection latency, so tune it per symptom: a hard-down service might use for: 2m, a slow latency creep for: 15m.

Severity is a plain label, not a built-in feature. The convention is severity: critical for “wake someone up” and severity: warning for “look at it during business hours”. The label carries no meaning to Prometheus itself; it becomes useful in Alertmanager routing, where you send critical to the pager and warning to a chat channel. Getting the split right is most of what separates a calm on-call rotation from a burned-out one. If everything is critical, nothing is.

A bad rule rewritten into a good one

Here is a rule that looks reasonable and pages constantly:

groups:
  - name: bad.rules
    rules:
      - alert: HighCPUUsage
        expr: rate(node_cpu_seconds_total{mode!="idle"}[5m]) > 0.8
        labels:
          severity: critical
        annotations:
          summary: "CPU is high"

Everything about it invites fatigue. It fires per core rather than per node, has no for so a single spike pages, calls a resource-usage number critical, and the annotation tells the responder nothing they can act on. Rewritten around a symptom:

groups:
  - name: good.rules
    rules:
      - alert: HighRequestErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
          sum(rate(http_requests_total[5m])) by (service)
            > 0.05
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.service }} is serving >5% errors"
          description: "5xx error ratio is {{ $value | humanizePercentage }} over 5m. Check recent deploys and downstream dependencies."
          runbook_url: "https://runbooks.internal/high-error-rate"

The good rule alerts on what a user experiences (a failed request), is aggregated to a meaningful unit (per service), survives a transient blip (for: 10m), and hands the responder a starting point through its annotations and a runbook link. Deciding where to set that 5% threshold is a job for an error budget and a defined service level objective rather than a number you invent at the keyboard. And this rule assumes the right metrics already exist: the error rate and latency queried here come from an application instrumented to expose them, a prerequisite covered in instrumenting an app for Prometheus.

Pitfalls that survive good intentions

Three traps recur even in symptom-based setups. First, the divide-by-zero: a ratio like the one above produces no result when a service gets no traffic, so an outage that stops all requests can silence the very alert meant to catch it. Pair error-rate alerts with a low-traffic or absence check. Second, over-eager thresholds without burn-rate awareness: a flat “5% for 10m” both misses a slow burn that quietly eats the month’s budget and screams during a brief spike, which is why multi-window, multi-burn-rate alerting exists. Third, critical inflation: every alert added as critical because “it felt important” trains the on-call to ignore the pager, and a muted pager is worse than no pager.

What to keep

Alert on symptoms, keep causes for diagnosis. Use for to filter self-healing noise, and reserve critical for things that genuinely need a human out of bed. Write annotations for the tired person reading them at 3am, not for yourself writing them at 3pm. An alerting stack built this way is a core deliverable of a reliability and observability engagement. More in the observability category, and the same discipline applied to probes lives in Kubernetes readiness and liveness probes.