cd ..

$ cat ~/field-notes/distributed-tracing-opentelemetry.md

Distributed Tracing with OpenTelemetry: Finding the Slow Service

5 min read 0x.run

The API request took 2.8 seconds. The API's own work took 40 milliseconds. Logs show successful calls to three downstream services, each with a different request ID.

This is the exact problem distributed tracing solves: reconstructing one operation as it crosses process boundaries.

Traces and Spans

A trace represents one end-to-end operation. A span represents one timed step inside it.

Trace: checkout request (1.24s)
│
├─ HTTP POST /checkout (1.24s)
│  ├─ validate cart (8ms)
│  ├─ SELECT inventory (32ms)
│  ├─ POST payment-service/charge (910ms)
│  │  ├─ token lookup (14ms)
│  │  └─ payment provider request (870ms)
│  └─ publish order.created (21ms)

Each span has a name, start and end time, status, attributes, events, and references to its parent. All spans share a trace ID. Every span has its own span ID.

The waterfall makes the critical path visible. A slow child explains the parent latency without correlating timestamps manually across five log systems.

Context Propagation Is the Important Part

Instrumentation can create beautiful local spans and still fail to build a distributed trace. The trace context must cross every boundary.

For HTTP, a caller injects context into request headers and the server extracts it. The standard traceparent shape carries version, trace ID, parent span ID, and flags:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Instrumentation libraries usually handle this automatically for supported HTTP clients and servers.

Queues need the same treatment. Inject trace context into message attributes when publishing, then extract it when consuming. Do not put it only in process-local memory; the consumer may run minutes later on another machine.

Start with Automatic Instrumentation

OpenTelemetry provides a vendor-neutral API, SDKs, and instrumentation ecosystem. Automatic instrumentation can cover framework requests, HTTP clients, database drivers, and common queue libraries.

That quickly produces infrastructure spans:

HTTP GET /users/:id
postgres SELECT users
HTTP GET profiles.internal
redis GET user:42

Auto-instrumentation is the baseline, not the finish line. It sees library calls but does not know that several calls together mean "reserve inventory" or "calculate shipping quote."

Add Manual Domain Spans

import { SpanStatusCode, trace } from '@opentelemetry/api';

const tracer = trace.getTracer('checkout-service');

export async function reserveInventory(order) {
    return tracer.startActiveSpan('inventory.reserve', async (span) => {
        try {
            span.setAttributes({
                'app.order.item_count': order.items.length,
                'app.inventory.strategy': 'all_or_nothing',
            });

            const result = await inventory.reserve(order.items);
            span.setAttribute('app.inventory.reserved_count', result.length);
            return result;
        } catch (err) {
            span.recordException(err);
            span.setStatus({
                code: SpanStatusCode.ERROR,
                message: err.message,
            });
            throw err;
        } finally {
            span.end();
        }
    });
}

Use stable operation names. A span named inventory.reserve groups cleanly. A span named reserve order 7f91c... for alice@example.com creates unbounded names and leaks sensitive data.

Attributes Need a Cardinality Budget

Attributes make traces searchable and aggregatable:

http.request.method = POST
http.response.status_code = 503
db.system.name = postgresql
app.checkout.payment_method = card
deployment.environment.name = production

Good attributes come from bounded sets. User IDs, order IDs, full URLs, SQL parameters, email addresses, and raw error bodies have high cardinality or sensitive content.

A trace already identifies one request. Add a business identifier only when operators truly need it, the backend can afford it, and privacy policy allows it. Hashing an email does not necessarily make it anonymous.

Events vs Child Spans

Use a child span when an operation has meaningful duration, attributes, or nested work. Use a span event for a point in time.

span.addEvent('inventory reservation retry', {
    'app.retry.attempt': 2,
    'app.retry.reason': 'lock_timeout',
});

Do not turn every log line into a span. Thousands of tiny spans make traces expensive and unreadable.

Put Trace IDs in Logs

Traces explain timing; logs retain detailed events. Correlate both.

const activeSpan = trace.getActiveSpan();
const context = activeSpan?.spanContext();

logger.info({
    trace_id: context?.traceId,
    span_id: context?.spanId,
    order_id: order.id,
}, 'inventory reserved');

Logging libraries can often inject active trace context automatically. The goal is bidirectional navigation: click from a trace span to its logs, or paste a log's trace ID into the trace search.

Do not generate a second "correlation ID" at each service if the trace ID already fulfills that role.

Sampling Controls Cost

Recording and exporting every span may be too expensive at high volume.

Head sampling decides near the start of the trace. It is cheap, but it does not yet know whether the request will become slow or fail.

Tail sampling decides after observing completed spans. It can retain errors, rare routes, and slow traces while discarding routine successes, but it requires buffering and a collector that sees the whole trace.

A practical policy might retain:

100% of errors
100% of requests slower than the service threshold
100% of explicitly selected critical operations
1-10% of routine successful traffic

Sampling decisions should propagate. If each service independently samples, traces become broken collections of unrelated fragments.

Use a Collector

Applications can export telemetry to an OpenTelemetry Collector instead of directly to a storage vendor.

applications
    ↓ OTLP
collector
    ├─ batch
    ├─ filter/redact
    ├─ retry
    ├─ sample
    └─ export to tracing backend

The collector centralizes buffering, retries, redaction, and vendor export configuration. It also keeps observability backend credentials out of every application process.

Telemetry must never block the request path indefinitely. Use bounded queues, short export timeouts, and drop telemetry under extreme pressure rather than taking down the service being observed.

Common Broken Traces

every service is a separate trace
  → propagation headers were dropped

one giant span for the entire request
  → add dependency and domain spans

millions of unique span names
  → IDs or raw URLs are embedded in names

successful HTTP span around a failed operation
  → application error status was never recorded

traces vanish during an incident
  → exporter queue is unbounded, blocked, or undersized

queue consumer appears as child of a week-old producer
  → consider a span link for asynchronous work instead of a direct parent

Instrumentation quality is observable too. Track export failures, dropped spans, collector queue depth, and sampling rates.

The Bottom Line

Distributed tracing connects work across processes into one timed causal graph.

The rules:

  • Propagate trace context across HTTP, queues, and background jobs
  • Start with automatic instrumentation, then add domain spans
  • Keep span names stable and attributes low-cardinality
  • Record exceptions and meaningful application failures
  • Correlate logs with trace and span IDs
  • Sample deliberately, retaining errors and slow traces
  • Use a collector for batching, redaction, retries, and export
  • Bound telemetry queues so observability cannot take down the service

The best trace is not the one with the most spans. It is the one that answers "where did the time go?" in under a minute.