A pod runs fine, then vanishes with OOMKilled and exit code 137. A different pod is slow under load while the node it sits on is 70% idle. Both symptoms trace back to the same two fields most manifests set by reflex and few people read closely: requests and limits. They are not two dials for the same thing. They act at different moments, on different subsystems, with very different consequences when you cross them.

Requests and limits do different jobs

A request is a scheduling input. The scheduler bin-packs pods by summing their requests against a node’s allocatable capacity; a request is a reservation and the floor the pod is guaranteed to get. It never caps usage.

A limit is a runtime ceiling enforced by the kubelet and the kernel through cgroups. What happens when a container hits that ceiling depends entirely on which resource you are talking about, because CPU and memory behave in opposite ways.

Crossing the CPU limit: throttling

CPU is a compressible resource. The kernel can hand a process fewer cycles without killing it.

The CPU request becomes a cgroup weight (cpu.shares under cgroup v1, cpu.weight under v2, where one core maps to 1024 shares). That weight only matters under contention: it decides how idle-starved processes split a saturated CPU. When the node has spare capacity, everyone runs freely regardless of requests.

The CPU limit becomes CFS bandwidth control: a quota per scheduling period (cpu.cfs_quota_us / cpu.cfs_period_us, or cpu.max on v2), with a default 100 ms period. A limit of 500m grants 50 ms of CPU per 100 ms window. Spend that budget before the window closes and the container is throttled, parked until the next period. No kill, just latency.

The trap lives in that per-period enforcement. Throttling fires whether or not the node is busy, so a bursty, latency-sensitive workload can be throttled hard while the machine sits mostly idle. Watch the ratio of container_cpu_cfs_throttled_periods_total to container_cpu_cfs_periods_total; a high value is CFS throttling, not a slow application.

Crossing the memory limit: OOMKill

Memory is incompressible. You cannot ask a process to hold less RAM the way you can give it fewer cycles.

The memory request feeds scheduling and eviction ranking; it is not a cap. The memory limit is a hard cap (memory.max / memory.limit_in_bytes). Touch one byte past it and the cgroup OOM killer sends SIGKILL to a process inside the cgroup. Kubernetes reports the container as OOMKilled, exit code 137 (128 + SIGKILL), restarts it per restartPolicy, and lands in CrashLoopBackOff if it keeps happening.

Two distinct memory events are easy to confuse. Hitting your own container’s limit is a cgroup OOMKill: kernel-level, immediate, scoped to that pod. The node running low on memory overall triggers kubelet eviction instead, which ranks victims by QoS class and by how far each pod has grown above its request.

QoS classes are derived, not declared

Kubernetes assigns each pod a class from what you set:

  • Guaranteed: every container sets requests == limits for both CPU and memory. It gets a strongly negative oom_score_adj (-997) and is evicted last.
  • Burstable: at least one request or limit is set, but the pod is not Guaranteed.
  • BestEffort: nothing set at all. It gets oom_score_adj 1000 and is the first to be evicted and the first the kernel kills under memory pressure.

The class is not cosmetic. Under node pressure it decides who dies first, and that ordering is the whole point of setting requests honestly.

The “no CPU limits” argument

This is a genuine, ongoing debate. The case for dropping CPU limits: CFS throttling costs latency even on an idle node, and a CPU request already guarantees a proportional share under contention, so a limit often just buys you throttling you never needed. The case for keeping them: predictable capacity planning, protection from a runaway neighbour, and the fact that CPU limits are required for the Guaranteed class and for static CPU pinning.

The pragmatic middle most platform teams land on: almost always set memory request == limit (incompressible resource, you want a hard and predictable cap), always set a CPU request, and treat CPU limits as optional, applied only where a specific policy demands one.

A method for setting defaults

Measure, do not guess. Use the Vertical Pod Autoscaler in recommendation mode (updateMode: "Off") or query Prometheus history directly. The same measure-first reflex applies to metric cardinality: observe before you tune, not the other way round.

# peak memory working set over 7 days, per pod
max_over_time(container_memory_working_set_bytes{container="app"}[7d])

# 95th percentile of CPU cores used over 7 days
quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{container="app"}[5m])[7d:5m])

Then set values from those numbers:

resources:
  requests:
    cpu: 250m           # typical usage, drives scheduling
    memory: 512Mi
  limits:
    memory: 512Mi       # equal to request: hard, predictable cap
    # no cpu limit: bursts into idle capacity, still guaranteed 250m under contention

Memory request == limit set to the observed peak working set plus roughly 20 to 30% headroom. CPU request at typical usage (P90-ish), CPU limit left off unless a policy needs one. Re-measure once real traffic lands; your first numbers are a hypothesis, not a verdict.

Common pitfalls

Setting only a limit makes Kubernetes copy it into the request, which can silently over-reserve the node. Copying another team’s numbers ignores that requests are workload-specific: too high wastes nodes, too low overcommits and invites eviction. Sizing memory off RSS instead of working set misses page cache the kernel counts against your limit. And blaming the application for latency that is actually CFS throttling wastes days; check the throttling ratio before you profile code.

What to remember

Requests are for the scheduler, limits are for the kernel. Over the CPU limit you get throttled, over the memory limit you get killed. Set memory request == limit, set a CPU request, be deliberate about CPU limits, and derive every number from measured usage rather than a template. Restarts from an OOMKill and restarts from a failing liveness probe look identical in kubectl describe, so it is worth knowing how probes actually behave before you debug the wrong one. Getting resource sizing right across a whole fleet is a good chunk of what makes a move to Kubernetes hold up in production instead of paging you later, and it is the heart of my legacy-to-Kubernetes migration work (in French). More Kubernetes material in the kubernetes category.