An admission webhook inserts a synchronous network call into the critical path of every Kubernetes object creation or modification: the API server waits for its response before accepting anything. That control point is powerful, and that’s exactly what makes it a single point of failure the moment its reliability hasn’t been engineered to match the API server’s own.

Two types, a precise execution order

A mutating webhook runs first and can rewrite the object before it’s persisted: injecting a sidecar, setting a default value, forcing a label. A validating webhook runs afterward, on the already-mutated object, and can only accept or reject, never modify. This ordering isn’t arbitrary: it guarantees validation applies to the final object, as it will actually be persisted, not to its pre-mutation original.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: require-signed-images
webhooks:
  - name: verify.signatures.example.com
    clientConfig:
      service:
        name: policy-controller
        namespace: policy-system
    rules:
      - apiGroups: [""]
        resources: ["pods"]
        operations: ["CREATE"]
    failurePolicy: Fail
    timeoutSeconds: 5

failurePolicy: the trade-off that decides everything

failurePolicy: Fail rejects the request if the webhook is unreachable or exceeds its timeoutSeconds: maximum security, but a down webhook then blocks every deployment triggering its rule, potentially including its own update if it goes through the same path. failurePolicy: Ignore does the opposite: the request gets admitted despite the webhook’s failure, preserving cluster availability at the cost of a control silently bypassed at the exact moment it would have mattered most.

# Check the webhook's real-world latency before choosing
# Fail: a p99 close to timeoutSeconds is a warning sign
kubectl get --raw /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations

There’s no universal correct setting: a webhook blocking Secret creation in production justifies Fail (the risk of a non-compliant Secret outweighs a temporary block); a cosmetic webhook on non-critical resources justifies Ignore. The decision gets made resource by resource, never by default.

The timeout, underestimated until the incident

timeoutSeconds (5 seconds by default, 30 at most) needs to sit well below the webhook service’s real response time, not equal to its observed average: an average of 2 seconds with a distribution tail climbing to 6 seconds under load exceeds a 5-second timeout exactly when the cluster is under the most pressure, usually the worst possible moment to start losing deployments. A webhook that’s slow under normal load becomes a webhook that fails under heavy load, a coupling that only surfaces on the fringe of a larger incident.

ValidatingAdmissionPolicy: the network-free alternative

Since Kubernetes 1.30, ValidatingAdmissionPolicy lets you express validation rules directly inside the API server via CEL (Common Expression Language), with no network call to an external service. The gain is structural: no network latency, no external point of failure, no failurePolicy trade-off to make for rules that fit this model.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-run-as-non-root
spec:
  validations:
    - expression: "object.spec.securityContext.runAsNonRoot == true"
      message: "Pods must set runAsNonRoot: true"

The trade-off: CEL stays more limited than a full webhook written in a general-purpose language, insufficient for validation logic that needs to call an external service (checking a signature against a registry, for instance). The two mechanisms coexist: ValidatingAdmissionPolicy for self-sufficient rules, a classic webhook for those that genuinely need an external call.

Takeaway

An admission webhook inserts a synchronous network call into the API server’s critical path, a powerful control point that becomes a single point of failure the moment its reliability doesn’t match the API server’s own. failurePolicy gets decided resource by resource, never by default; timeoutSeconds needs to cover the service’s real distribution tail, not its average. ValidatingAdmissionPolicy eliminates that risk for self-sufficient rules expressible in CEL, a mechanism worth knowing before adding one more webhook to a cluster’s critical path during a CI/CD industrialization.