Your cluster is covered in metrics. node_exporter reports CPU, RAM and disk usage, kube-state-metrics knows the state of every pod. Then a product manager asks “is checkout slower than this morning?”, and nobody can answer. All of those metrics describe the machines and the orchestrator; none of them describe what your application does. That is exactly the gap instrumentation fills: exposing, from the code, the few numbers that say whether the service is serving.

The four signals, and where to start

The most useful grid is still the golden signals from Google’s SRE book: latency (how long a request takes), traffic (how much you get), errors (how much fails), and saturation (how loaded your resources are). Saturation is already well covered by node_exporter and cluster metrics. That leaves three things to instrument in the application itself, which the RED method captures neatly for anything that answers requests: Rate, Errors, Duration.

In other words, before adding a single business metric, expose how many requests arrive, how many go wrong, and how long they take. Everything else is diagnosis you reach for once one of those three signals has paged you.

The right metric type for the question

Prometheus has four metric types, and the choice is not cosmetic. A counter only ever goes up and is read with rate(): it is the type for traffic and errors, an http_requests_total carrying a status label. A gauge is an instantaneous value that goes up and down, for anything counted at a point in time (in-flight connections, queue depth). A histogram samples a distribution into predefined buckets, and it is the right tool for latency. A summary looks similar but computes its quantiles differently, with a heavy consequence covered below.

An instrumented handler, and the queries to match

Concretely, your client library exposes a text format like this at scrape time:

# TYPE http_requests_total counter
http_requests_total{service="checkout",status="200"} 128934
http_requests_total{service="checkout",status="500"} 271

# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{service="checkout",le="0.1"} 24054
http_request_duration_seconds_bucket{service="checkout",le="0.3"} 33444
http_request_duration_seconds_bucket{service="checkout",le="1"}   34101
http_request_duration_seconds_bucket{service="checkout",le="+Inf"} 34115
http_request_duration_seconds_sum{service="checkout"}   5342.7
http_request_duration_seconds_count{service="checkout"} 34115

The histogram exposes cumulative counters per le (less than or equal) boundary, plus a _sum and a _count. From there, two queries cover most of the ground:

# error rate per service
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  /
sum(rate(http_requests_total[5m])) by (service)

# p99 latency per service
histogram_quantile(
  0.99,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

These are precisely the metrics that symptom-based Prometheus alerts consume: instrument first, alert second. A rule watching error rate or latency only exists because the application was instrumented to expose them.

Histogram or summary: why the quantile is computed server-side

Here is the distinction that decides the type. A summary computes its quantiles in the client, per instance, and publishes a ready-made quantile="0.99". The catch: you cannot sum or average quantiles. The p99 of ten pods is not the mean of their ten p99s. The moment your service runs in more than one replica, a summary hands you ten numbers that cannot be recombined into one service latency.

A histogram does the opposite: it exposes raw per-bucket counts, and histogram_quantile() reconstructs the quantile server-side, after you aggregate the buckets with sum by (le). Since everything runs in multiple copies on Kubernetes, the histogram is the default. Its cost: you have to choose the buckets up front.

Buckets, interpolation, and the pitfalls

histogram_quantile() interpolates linearly within the bucket where the quantile lands. The precision of your p99 is therefore the width of the bucket around it. The default buckets (a few milliseconds to ten seconds) rarely bracket a real target, so place the boundaries where you make decisions. If your target is 300ms, you want boundaries near 0.25, 0.3, 0.5. And if the quantile falls in the +Inf bucket, the result stops being a value and becomes a lower bound, a sign your buckets are too coarse. Native histograms, a newer option, remove this bucket choice at the price of still-experimental status.

Three other traps recur. Cardinality first: a label per user, per raw URL, or per request ID multiplies your series by the number of buckets and blows up storage cost, a subject the glossary entry on metric cardinality covers. Keep labels bounded: route template, not raw path; status class, not full message. Averaging quantiles second, an avg of a p99, which means nothing: aggregate the buckets, never the quantiles. Too few buckets around the target last, which makes the p99 unreadable at the exact moment it matters.

What to keep

Instrument three things from the code, traffic, errors and duration, and leave saturation to node_exporter. A counter for rates, a histogram for latency, with its boundaries placed around the decision you actually make. Prefer histograms to summaries the moment the service runs in more than one replica, and keep label cardinality bounded, because an instrumented app that costs more than it observes is its own kind of outage. This kind of useful observability is the foundation of my reliability and observability work; more in the observability category.