etcd stores the entire state of a Kubernetes cluster: every object, every Secret, every ConfigMap, every scheduler decision passes through it before existing anywhere else. It’s the most critical component of the control plane and, paradoxically, the one that gets the least attention, because it works silently right up until the day it doesn’t.

What actually depends on etcd

The Kubernetes API server only reads from and writes to etcd: every object creation, modification, or deletion goes through an etcd write before being considered effective. An unreachable etcd doesn’t crash pods already running (the kubelet keeps running what it already knows about), but it blocks everything depending on a new write: no new deployment, no scaling, no reconciliation from a controller that needs to read current state. A cluster with etcd down looks alive on the surface while being frozen underneath.

# Check the etcd cluster's health directly,
# not just the API server's, which partly masks it
etcdctl endpoint health --cluster

Disk latency, the one metric that actually matters

etcd uses Raft (see the article on Raft consensus) to replicate every write to a majority of nodes before confirming it, which implies a disk fsync on every confirmed write. A slow disk (non-dedicated network storage, an SSD shared with other workloads) turns that requirement into a bottleneck: etcd’s documentation sets an fsync latency threshold around 10ms beyond which the cluster becomes unstable, with spurious leader elections that have nothing to do with an actual network outage.

# The metric that predicts trouble before it turns
# into cascading leader elections
etcd_disk_wal_fsync_duration_seconds

etcd hosted on network storage without guaranteed IOPS, a common choice for infrastructure simplicity, is the most frequent cause of control-plane instability that has nothing to do with Kubernetes itself.

Backups, never optional

An etcd backup (etcdctl snapshot save) captures the cluster’s complete state at a point in time, the only recovery path in case of quorum loss (a majority of etcd nodes unavailable simultaneously, a poorly distributed datacenter outage, for instance). One distinction is worth getting right: what is irrevocably lost is the quorum, not necessarily the data. As long as one member’s data directory survives, restarting that member with --force-new-cluster overwrites cluster membership while keeping existing application data. The etcd documentation strongly discourages the move: it panics if any member of the previous cluster is still alive, and it rebuilds a single-node cluster from a state nothing guarantees was current at the time of the outage. It is an emergency exit, not a restore plan. If no data directory is usable anymore, though, then no command rebuilds a state that was never captured anywhere else.

# Regular backups, automated and tested,
# not written once and forgotten
etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db

# Verification is a local operation on the file,
# so it belongs to etcdutl. In etcdctl, "snapshot status"
# was deprecated in 3.5, then removed in v3.6.
etcdutl snapshot status /backup/etcd-snapshot-$(date +%Y%m%d).db

A backup never restored under real conditions is just a hypothesis: the only way to know a restore actually works is to have already practiced it, on a test cluster, before needing it in an emergency.

Node count, a choice made once

A 3-node etcd cluster tolerates losing one node without losing quorum; a 5-node cluster tolerates two, at the cost of wider replication and slightly higher write latency (more nodes must confirm each write). An even number structurally adds nothing: 4 nodes offer no more fault tolerance than 3, while doubling the risk of a deadlock during a 50/50 network partition, exactly the reasoning detailed in the article on Raft.

Takeaway

etcd carries a Kubernetes cluster’s entire state, but its unavailability doesn’t kill already-running pods: it silently freezes everything depending on a new write, an easy symptom to miss before it escalates. Disk latency (fsync) is the metric that predicts instability before cascading leader elections; a backup never tested via restore is just an unverified hypothesis. These three habits (watching disk latency, backing up and testing restores, choosing an odd node count) are the quiet foundation any durable Kubernetes migration rests on.