A service registered in Consul and a service actually able to answer a request are two different things, and confusing the two is the most common cause of traffic being routed to a dead instance. Consul does not stop at knowing who exists: it continuously checks who answers, and it is that check, not the registration itself, that makes the system worth anything.

The catalog: who exists, not who is healthy

When a service starts, a Consul agent (running on every node) registers it in the catalog with its address, its port, and optional metadata. The catalog on its own says nothing about the health of the service: an instance that crashed without deregistering cleanly stays visible in it indefinitely.

service {
  name = "payment-api"
  port = 8080
  check {
    http     = "http://localhost:8080/health"
    interval = "10s"
    timeout  = "2s"
  }
}

Health checks: what makes the catalog trustworthy

It is the check block that turns a plain directory into a reliable discovery system. Consul periodically queries every registered service and removes from query results any instance that fails its check, without ever deleting it from the catalog itself: the distinction between “registered” and “healthy” stays visible, which helps diagnosis (a critical instance is not a missing instance).

Three check types exist, with different guarantees. An http check queries an endpoint and verifies the status code, which confirms that the process answers but nothing more. A script check runs a local command, useful for specific verifications (disk space, connection to a dependency) but it runs on the node, not from the outside. A ttl check inverts the logic: the service itself has to actively confirm its health at a regular interval, and the absence of confirmation counts as a failure, which is useful to detect a process that still exists but no longer makes progress (a deadlock, for example), something a plain HTTP ping would never see.

The shallow health check trap

An http check that only verifies that GET /health returns a hardcoded 200 OK tests nothing real: the process can answer while its critical dependency (database, cache) is unreachable, and the check stays green while the service fails silently on every actual request. A health check that is worth something verifies what the service actually needs in order to work, not just that a port is listening.

# Fake health check: tests nothing
GET /health -> always 200

# Useful health check: verifies the real dependency
GET /health -> 200 only if the DB connection answers

The opposite exists too, and costs just as much: a check that is too strict (failing on the slightest latency of a non-critical dependency) pulls perfectly functional instances out of routing, creating an artificial capacity shortage exactly when load is already high.

Why Consul needs consensus (Raft)

Consul servers (distinct from the agents running on each application node) have to agree among themselves on the state of the catalog, including during a partial failure of the cluster. Consul uses Raft, a consensus algorithm that requires a strict majority of servers (a quorum) to be available and in agreement for the cluster to accept writes. With 3 Consul servers, the cluster tolerates the loss of one without interruption; with 5, it tolerates two. An even number of servers brings no additional tolerance over the odd number immediately below it, while doubling the risk of a partition into two equal groups with no clear majority: this is why the Consul documentation systematically recommends an odd number of servers.

Where this fits

Consul answers the same need as service discovery in the broad sense, with an active health mechanism that native Kubernetes Services do not offer out of the box (they route to whatever is Ready, but the definition of Ready is left to your own probes). Choosing between the two, or running both together on a hybrid infrastructure, is part of the decisions made during a Kubernetes migration.

Takeaway

The Consul catalog says who exists, health checks say who actually answers, and confusing the two routes traffic to dead instances. A shallow check (one that only tests that a port is listening) protects nothing; a check that is too strict pulls healthy instances out. Raft consensus imposes an odd number of servers to guarantee a clear quorum during a partial failure.