Distributed Tracing

Trace Context

Trace context propagates trace information across service boundaries.

Context Elements

Trace Context:

1. Trace ID:
   - Unique identifier for entire request
   - Same across all services

2. Span ID:
   - Unique identifier for single operation
   - Different for each service

3. Parent Span ID:
   - Link to parent operation
   - Creates hierarchy

4. Trace Flags:
   - Sampling decision
   - Debug flags

W3C Trace Context Header

Header: traceparent
Format: 00-<trace-id>-<span-id>-<trace-flags>

Example:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

00: version
4bf92f3577b34da6a3ce929d0e0e4736: trace ID
00f067aa0ba902b7: span ID
01: trace flags (sampled)

Propagation Example

import requests
from opentelemetry import trace
from opentelemetry.propagate import inject, extract

def service_a():
    with tracer.start_as_current_span('service_a_operation') as span:
        # Inject context into headers
        headers = {}
        inject(headers)
        
        # Call service B with context
        response = requests.get('http://service-b/api', headers=headers)

def service_b():
    # Extract context from headers
    context = extract(request.headers)
    
    with tracer.start_as_current_span('service_b_operation', context=context) as span:
        # This span is child of service_a's span
        process_request()

Context Propagation Methods

Method	Description	Use Case
HTTP Headers	W3C traceparent	REST APIs
gRPC Metadata	Binary propagation	gRPC services
Message Headers	Custom headers	Message queues
URL	Query parameter	Redirects

Best Practices

1. Use W3C standard for HTTP
2. Propagate context across all boundaries
3. Don't break the chain
4. Sample consistently
5. Handle missing context gracefully

Span Hierarchy

Span Structure

Trace: abc123

Span 0: api-gateway (180ms)
├── Span 1: auth-service (20ms)
├── Span 2: order-service (150ms)
│   ├── Span 3: validate-order (10ms)
│   ├── Span 4: inventory-service (50ms)
│   │   └── Span 5: db-query (30ms)
│   ├── Span 5: payment-service (70ms)
│   │   └── Span 6: stripe-api (60ms)
│   └── Span 7: db-insert (15ms)

Span Attributes

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span('process_order') as span:
    # Set attributes
    span.set_attribute('order.id', '123')
    span.set_attribute('order.amount', 99.99)
    span.set_attribute('user.id', '456')
    
    # Add events
    span.add_event('validation_started')
    validate_order()
    span.add_event('validation_completed')
    
    # Set status
    span.set_status(trace.StatusCode.OK)

Span Kinds

Kind	Description	Example
INTERNAL	Within service	Business logic
SERVER	Incoming request	HTTP server
CLIENT	Outgoing request	HTTP client
PRODUCER	Message producer	Kafka producer
CONSUMER	Message consumer	Kafka consumer

Span Timing

Span Timeline:

|←─────── Parent Span ────────→|
     |←── Child 1 ──→|
                       |←── Child 2 ──→|
                              |←─ Grandchild ─→|

Start time: When operation begins
End time: When operation completes
Duration: End - Start

Best Practices

1. Name spans by operation
2. Add meaningful attributes
3. Record errors in spans
4. Keep span depth reasonable
5. Don't over-instrument

Jaeger and Zipkin

Jaeger

Jaeger Architecture:

Agent → Collector → Query → Storage
  ↑                     ↓
Services              UI

- Open source (Uber)
- OpenTelemetry native
- Adaptive sampling

Jaeger Implementation

from jaeger_client import Config
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.jaeger.thrift import JaegerExporter

# Setup Jaeger
jaeger_exporter = JaegerExporter(
    agent_host_name='localhost',
    agent_port=6831,
)

# Create tracer
provider = TracerProvider()
processor = BatchSpanProcessor(jaeger_exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

Zipkin

Zipkin Architecture:

Reporter → Collector → Storage → API → UI
  ↑                              ↓
Services                       Query

- Open source (Twitter)
- Simple setup
- B3 propagation

Zipkin Implementation

from opentelemetry.exporter.zipkin import ZipkinExporter

zipkin_exporter = ZipkinExporter(
    endpoint='http://localhost:9411/api/v2/spans'
)

provider = TracerProvider()
processor = BatchSpanProcessor(zipkin_exporter)
provider.add_span_processor(processor)

Comparison

Feature	Jaeger	Zipkin
Origin	Uber	Twitter
OpenTelemetry	Native	Supported
Sampling	Adaptive	Rate-based
Storage	Cassandra, ES	Cassandra, ES, MySQL
UI	Feature-rich	Simple
Complexity	Higher	Lower

When to Use

Scenario	Recommendation
OpenTelemetry native	Jaeger
Simple setup	Zipkin
Adaptive sampling	Jaeger
Existing infrastructure	Match existing

Best Practices

1. Start with Jaeger or Zipkin
2. Use OpenTelemetry SDK
3. Implement adaptive sampling
4. Monitor trace storage
5. Set retention policies