A configured PodDisruptionBudget, a tuned liveness probe, an HPA that scales: each of these mechanisms rests on an assumption verified exactly once, in theory, when it was written. Chaos engineering replaces that assumption with proof: deliberately triggering the failure, under controlled conditions, to check the system reacts as expected, before a real incident discovers otherwise on its behalf.

A discipline, not a tool you install

Chaos engineering isn’t primarily a tooling question: it’s an experimental method. Formulate a precise hypothesis (“if this pod dies, traffic shifts to remaining replicas with no visible user-facing error”), define a limited blast radius, run the experiment, observe whether the hypothesis holds. A tool like Litmus or Chaos Mesh automates execution, but the discipline comes before the tool: without a clear hypothesis, a randomly triggered failure teaches no more than a real incident, and worse (nobody was on alert for it).

# Chaos Mesh: kill a random pod from the deployment,
# a minimal but representative experiment
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: kill-checkout-pod
spec:
  action: pod-kill
  mode: one
  selector:
    labelSelectors:
      app: checkout

Start small, in a controlled environment

The first chaos experiment never runs in production without preparation: a staging environment representative of real load and topology is the starting point, with a deliberately narrow blast radius (a single pod, a single availability zone) before considering anything larger. The initial goal isn’t to break things hard, it’s to check that observability itself works: a triggered failure that generates no alert reveals a more urgent problem than the failure being tested.

The experiments that reveal the most

Killing a pod validates the most basic mechanism (automatic replacement), but the experiments that reveal the most are often less obvious: injecting network latency toward a dependency (tests timeouts and retries, not just availability), exhausting a node’s memory (tests eviction priorities and PriorityClass), or simulating the loss of an entire availability zone (tests pod anti-affinity as actually configured, not just documented).

# Inject 200ms of latency toward a dependent service,
# reveals mis-tuned timeouts a simple kill wouldn't catch
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
spec:
  action: delay
  delay:
    latency: "200ms"
  selector:
    labelSelectors:
      app: payment-gateway

This kind of experiment regularly surfaces defects no code review catches: a client timeout configured shorter than the server timeout, a dependency with no circuit breaker cascading slowness downstream, a retry with no backoff making an overload worse instead of absorbing it.

The game day: the organized version, not improvised

A game day formalizes the exercise: a chaos experiment planned in advance, with the on-call team actively watching (not surprised), a precise learning objective, and a debrief documenting what was actually discovered. That formalization turns chaos engineering from a one-off exercise into a repeatable practice, aligned with an SLO’s review cycle: every game day tests one precise resilience hypothesis, not a generic failure.

Takeaway

A resilience mechanism never actively tested (PodDisruptionBudget, liveness probe, retry) remains a hypothesis until proven otherwise, and the cheapest proof is a deliberately triggered experiment rather than a real incident. Start small, in staging, with a narrow blast radius and a precise hypothesis, before considering production; the most revealing experiments go beyond a simple pod kill (injected latency, resource exhaustion); the game day organizes this practice into a repeatable exercise rather than a one-off, one of the pillars of reliability and observability built on proof rather than assumption.