A request that crosses ten microservices and gets slow somewhere along the way tells you nothing from aggregated metrics (they show rising average latency, never which specific request or where) nor from logs (each service writes its own, with no explicit link to the others). Distributed tracing exists to answer a question neither can: this specific request, where did it spend its time?

A trace, made of spans

A trace represents a request’s complete journey through a system. It is made up of spans, each representing a unit of work: an HTTP call, a database query, an internal computation. Every span carries a start time, a duration, and a link to its parent span, which reconstructs the request’s full hierarchy.

Trace: a3f9c2e1...
├── span: API Gateway (45ms)
│   └── span: payment-service (38ms)
│       ├── span: fraud-check (12ms)
│       └── span: db-query INSERT (20ms)

That structure answers the original question directly: out of 45ms of total latency, 20ms comes from one specific database query, not from some diffuse slowdown impossible to pin down.

Context has to travel with the request

The mechanism that makes all of this possible is context propagation: every service receiving a request must forward the trace identifier to whichever service it calls next, typically through a standardized HTTP header (traceparent, defined by the W3C Trace Context standard). A service that forgets to propagate that header breaks the chain: downstream spans start a brand new trace with no visible link to what came before, making the problem invisible exactly where it would be most useful to see it.

traceparent: 00-a3f9c2e1b8d4f012-9c2e1b8d4f012a3f-01
             │  └─ trace-id ─────┘└─ span-id ────┘ │
             version                              flags

What OpenTelemetry standardizes

Before OpenTelemetry, every tracing backend (Jaeger, Zipkin, proprietary solutions) imposed its own instrumentation SDK, which locked application code into a vendor choice. OpenTelemetry separates instrumentation (the SDK generating spans in the code) from the backend that stores and displays them: an OTel Collector receives traces in the standard OTLP format and can route them to Jaeger, Tempo, or a commercial backend, without ever touching the instrumented application code.

# Minimal example: export traces to a local Collector
exporters:
  otlp:
    endpoint: "otel-collector:4317"

The sampling trap

Tracing every single request, with the full volume of spans that represents, quickly becomes expensive to store and process at scale. Sampling reduces that volume by keeping only a fraction of traces, but naive sampling (keep one request in a hundred, at random) has a counter-intuitive consequence: the most interesting requests, the ones that fail or run abnormally slow, are statistically the most likely to get dropped, since they’re rare by definition.

Tail-based sampling fixes this by deciding whether to keep a trace after seeing it in full, not randomly at its start: a trace containing an error or exceeding a latency threshold gets kept every time, while normal traffic stays sampled to control volume. The trade-off: this requires buffering in-flight traces before deciding, which costs more resources at the Collector level than sampling at the entry point.

Where this fits

Tracing complements golden signals and centralized logs rather than replacing them: metrics say a problem exists, logs say what happened at a precise moment, traces say where time got lost across a full journey. Setting up this third layer is part of what a reliability and observability engagement covers.

What to remember

A trace is made of hierarchical spans that reconstruct a request’s full journey; context must be explicitly propagated from one service to the next, or the chain breaks silently. OpenTelemetry standardizes instrumentation independently of the storage backend. Naive sampling statistically drops the most useful traces (errors, abnormal latency); tail-based sampling fixes that bias at the cost of extra processing resources.