The Linux kernel tracks every active network connection (TCP, stateful UDP, ICMP) in an in-memory table, conntrack, required for NAT and stateful firewalling. This table has a maximum size, nf_conntrack_max, and what happens once that limit is reached is almost never what you’d expect: no gradual slowdown, an immediate and silent rejection of new connections.

What conntrack does, and why Kubernetes depends on it

Every connection passing through NAT (nearly all Kubernetes Service traffic) creates an entry in the conntrack table, associating the original connection with its NAT translation, needed to correctly route return packets. A cluster with a high volume of short-lived connections (lots of small HTTP requests, a high-throughput service) generates a proportional volume of conntrack entries, not just to simultaneous connections but to their creation rate.

# The current limit and the number of tracked entries,
# the only check that reveals an imminent problem
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

What actually happens at the limit

A full conntrack table doesn’t slow down new connections: the kernel rejects them immediately, a new connection attempt failing with a symptom that looks like a distant network problem, a blocking firewall, or an application refusing the connection, never explicitly “this node’s conntrack table is full.” The symptom shows up on the node, not on the application running on it, which often misdirects diagnosis toward the wrong component.

# The counter revealing rejections already in progress,
# concrete proof the limit has been reached
cat /proc/sys/net/netfilter/nf_conntrack_max
dmesg | grep "nf_conntrack: table full"

The Kubernetes-specific trap: systematic NAT

A Kubernetes cluster routes a disproportionate share of its traffic through NAT, even for purely internal pod-to-pod communication, depending on the CNI configuration and kube-proxy mode. That NATed traffic volume, invisible under normal operation, fills the conntrack table faster than an equivalent infrastructure without this network virtualization layer, a scaling factor often absent from a node’s initial sizing.

Sizing and monitoring, not just increasing

Increasing nf_conntrack_max pushes the problem back without ever solving it if connection volume keeps growing, and a value set too high consumes non-trivial kernel memory (each entry occupies a fixed space, multiplied by millions of entries on a high-traffic node).

# Raising the limit, a temporary fix
# if connection volume keeps growing
sysctl -w net.netfilter.nf_conntrack_max=524288

Continuously monitoring the nf_conntrack_count / nf_conntrack_max ratio (via Prometheus, node_exporter exposes it natively) turns a silent incident into an actionable alert before actual exhaustion, rather than discovering the limit the moment connections start failing.

Takeaway

A full conntrack table doesn’t degrade gradually: it immediately and silently rejects any new connection, a symptom that looks like a distant network or application problem, never explicitly the real cause. Kubernetes worsens this risk through systematically NATed traffic, even internally, a scaling factor often overlooked in initial sizing. Continuously monitoring the occupancy ratio, rather than raising the limit after the fact, turns a silent incident into an actionable signal before exhaustion, one of the low-level network details that matter the moment a Kubernetes migration runs high-connection-volume traffic.