The number of metrics declared in an application almost never sizes a Prometheus server: it’s cardinality, the number of unique time series actually stored, that decides. A single poorly chosen label can multiply that number by a thousand, without a single additional metric ever being added.

The multiplication that catches everyone off guard

Every distinct combination of metric name and label values creates an independent series. http_requests_total broken down by 10 routes, 5 status codes, and 20 pods doesn’t weigh one metric: it weighs 1,000 series, each with its own stored history.

# A single instrumented metric line...
http_requests_total{route, status, pod}

# ...becomes 1000 distinct series once combined:
# 10 routes × 5 status × 20 pods = 1000 active series

The classic trap: a label with unbounded values, like a request ID or a client IP address, where every new value creates one more series, indefinitely. A histogram makes the math worse still, since it emits one series per latency bucket on top of the dimension already multiplied by every other label.

# Trap: request_id has as many possible values
# as there are requests, cardinality never stops growing
http_requests_total.labels(route=route, status=status, request_id=request_id).inc()

Kubernetes makes it worse by construction

Normal pod churn on Kubernetes produces the same effect as an unbounded label, even with no instrumentation mistake at all: every redeploy generates new pod names, and therefore new series for any metric carrying a pod label. A deployment that redeploys several times a day (the common case with active CI/CD) accumulates dead series continuously, each kept in memory as long as Prometheus still considers it active, absent any new sample.

Cutting it at the source: metric_relabel_configs

The most cost-effective lever applies at scrape time, before the series ever enters the TSDB at all: dropping or collapsing an overly granular label directly in the Prometheus configuration.

scrape_configs:
  - job_name: checkout
    metric_relabel_configs:
      # Drops the request_id label before storage:
      # the series will never get created, whatever its volume
      - action: labeldrop
        regex: request_id

A label already instrumented in application code but never actually useful in practice gets handled the same way, no need to redeploy the application: the scrape configuration alone is enough.

The safety net: sample_limit

sample_limit caps the number of samples accepted per scrape target, a hard safeguard protecting the Prometheus server from a target that drifts suddenly (an instrumentation bug that abruptly generates thousands of label values), at the cost of rejecting the entire scrape once the limit is exceeded.

scrape_configs:
  - job_name: checkout
    sample_limit: 10000

That cap isn’t a fix for the underlying problem, it’s a levee: a fully rejected scrape beyond the limit beats a Prometheus server running out of memory, but the real fix stays addressing the cause (the unbounded label) at the source.

Pre-aggregating what matters, via recording rules

A PromQL query aggregating across thousands of series on every alert evaluation or every dashboard refresh burns an avoidable recurring cost: a recording rule pre-computes that aggregation once, on a regular interval, and stores the result as a single, far cheaper series to query afterward.

groups:
  - name: checkout-aggregations
    rules:
      - record: checkout:http_requests:rate5m
        expr: sum(rate(http_requests_total{job="checkout"}[5m])) by (route)

Measure before fixing blind

prometheus_tsdb_head_series gives the total active series at a given moment, the first metric to watch. To pinpoint exactly which metrics weigh the most, topk(10, count by (__name__)({__name__=~".+"})) ranks the biggest contributors, an essential step before choosing where to apply a labeldrop rather than guessing.

topk(10, count by (__name__)({__name__=~".+"}))

Takeaway

Cardinality, not the number of declared metrics, is what actually sizes a Prometheus server: a single unbounded-value label multiplies stored series with no natural ceiling, an effect Kubernetes amplifies through the plain normal churn of its pods. metric_relabel_configs fixes it at the source, before the series ever exists; sample_limit protects against a sudden drift without fixing its cause; recording rules avoid recomputing an expensive aggregation on every query. Measuring before fixing (prometheus_tsdb_head_series, ranking via topk) avoids guessing where to act, a discipline that’s part of the foundation of reliability and observability that holds up over time, not just at launch.