The OpenTelemetry Collector isn’t a single component but a two-tier architecture: the agent, deployed as close as possible to each application, and the gateway, a centralized tier that sees all traffic before deciding what to do with it. Conflating the two roles, or deploying only one by reflex, limits what observability can actually do.

The agent: one Collector per node, as close to the source as possible

Deployed as a DaemonSet (one pod per node), the agent receives traces and metrics directly from applications running on that same node, before any centralized processing. Its role stays local and limited: receive, apply light initial filtering, enrich with metadata (node name, pod name) available locally, then forward to the next tier.

# The DaemonSet agent only sees traffic
# from applications running on its own node
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-agent
spec:
  template:
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector

The gateway: a tier that sees everything, before anything gets decided

The gateway, deployed as a Deployment (a handful of replicas, not one per node), receives aggregated traffic from every agent. That centralization changes what becomes possible: a decision that needs to see an entire trace, potentially spread across several nodes and several services, can only be made at this tier, never at an agent that only sees a local fraction of the traffic.

# The gateway aggregates traffic from every agent,
# the only tier that sees a trace in its entirety
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-gateway
spec:
  replicas: 3

Why tail-sampling specifically requires the gateway

Tail-based sampling decides whether to keep a trace after seeing it in full (systematically keeping error or slow traces, sampling the rest), a decision that assumes having every span of a trace in one place before deciding. A trace crossing three services spread across three different nodes produces spans captured by three separate agents: no agent, isolated on its node, can apply correct tail-sampling, since none of them sees the trace in its entirety. Only the gateway, receiving the aggregated whole, has the information needed.

# Tail-sampling can only be applied at the gateway
# tier, never at an agent seeing only a fraction
processors:
  tail_sampling:
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}

The cost of this two-tier architecture

The gateway must buffer a trace’s spans until it’s complete before deciding, consuming memory proportional to the volume of concurrent traces and their lifespan. On a high-traffic system, that buffer becomes a sizing concern of its own, distinct from agent sizing (which only relays without waiting). Skipping the gateway tier and doing tail-sampling directly in the agents, a tempting infrastructure saving, produces incorrect sampling by construction: each agent only judges its local fraction, never seeing the whole.

Takeaway

OpenTelemetry’s Collector architecture separates two distinct roles: the agent (DaemonSet, one per node, local collection) and the gateway (Deployment, a few replicas, aggregated view of all traffic). Tail-sampling, which decides after seeing a complete trace, can only be applied correctly at the gateway tier, never at an agent seeing only a local fraction, an architectural detail that determines whether intelligent sampling actually works or only appears to. This architectural choice is part of the foundation of reliability and observability that genuinely holds up at scale, not just on a modest-traffic system.