Observability

The Three Pillars of Observability

Observability = Logs + Metrics + Traces

The Three Pillars

1. Logs:
   - Discrete events
   - Detailed information
   - Debugging individual requests

2. Metrics:
   - Numerical measurements
   - Aggregated over time
   - Trend analysis

3. Traces:
   - Request journey through system
   - Latency breakdown
   - Service dependency mapping

Comparison

Pillar	What	When	Granularity
Logs	Events/details	Debugging	Individual
Metrics	Aggregated numbers	Monitoring	Aggregated
Traces	Request flow	Performance	Per-request

How They Work Together

Problem: High latency detected

1. Metrics: Latency P99 = 2s (normally 100ms)
2. Traces: Find slow request, see which service is slow
3. Logs: Check that service's logs for errors

Metrics → Traces → Logs (drill down)

Observability Stack

Logs: ELK, Loki, Fluentd
Metrics: Prometheus, Grafana, Datadog
Traces: Jaeger, Zipkin, OpenTelemetry

All three: OpenTelemetry (unified)

Design for Observability

1. Structured logging
2. Correlation IDs
3. Standard metrics
4. Sampling strategies
5. Dashboards and alerts

Logs in Observability

Structured Logging

import json
import logging

class StructuredLogger:
    def __init__(self, service_name):
        self.service = service_name
    
    def log(self, level, message, **kwargs):
        log_entry = {
            'timestamp': datetime.utcnow().isoformat(),
            'level': level,
            'service': self.service,
            'message': message,
            'trace_id': get_trace_id(),
            'span_id': get_span_id(),
            **kwargs
        }
        print(json.dumps(log_entry))
    
    def info(self, message, **kwargs):
        self.log('INFO', message, **kwargs)
    
    def error(self, message, **kwargs):
        self.log('ERROR', message, **kwargs)

# Usage
logger = StructuredLogger('order-service')
logger.info('Order created', order_id='123', user_id='456')

Log Levels

Level	Use Case
DEBUG	Detailed debugging info
INFO	General operational events
WARN	Unexpected but handled
ERROR	Errors requiring attention
FATAL	System-threatening errors

Correlation

def correlation_middleware(next_app):
    def wrapper(request):
        # Generate or extract correlation ID
        correlation_id = request.headers.get('X-Correlation-ID') \
                        or str(uuid.uuid4())
        
        # Set in context
        set_correlation_id(correlation_id)
        
        # Add to response
        response = next_app(request)
        response.headers['X-Correlation-ID'] = correlation_id
        return response
    return wrapper

Best Practices

1. Use structured logging (JSON)
2. Include correlation IDs
3. Log at appropriate levels
4. Don't log sensitive data
5. Centralize logs

Metrics in Observability

Metric Types

1. Counter:
   - Monotonically increasing
   - Example: requests_total

2. Gauge:
   - Can go up or down
   - Example: temperature, queue_depth

3. Histogram:
   - Distribution of values
   - Example: request_duration

4. Summary:
   - Similar to histogram
   - Client-side calculation

Prometheus Example

from prometheus_client import Counter, Histogram, Gauge

# Define metrics
REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'endpoint'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency', ['endpoint'])
QUEUE_DEPTH = Gauge('queue_depth', 'Current queue depth')

# Use metrics
def handle_request():
    REQUEST_COUNT.labels(method='GET', endpoint='/api').inc()
    
    with REQUEST_LATENCY.labels(endpoint='/api').time():
        result = process_request()
    
    QUEUE_DEPTH.set(get_queue_depth())
    return result

Key Metrics

Category	Metrics
Traffic	requests_per_second, concurrent_users
Latency	p50, p95, p99 duration
Errors	error_rate, errors_by_type
Saturation	cpu_usage, memory_usage, queue_depth

SLIs/SLOs

SLI (Service Level Indicator):
- Latency P99 < 200ms
- Error rate < 0.1%
- Availability > 99.9%

SLO (Service Level Objective):
- 99.9% of requests < 200ms
- 99.95% success rate

SLA (Service Level Agreement):
- Contractual commitment
- Penalties for missing

Best Practices

1. Use RED method (Rate, Errors, Duration)
2. Use USE method (Utilization, Saturation, Errors)
3. Set SLOs and alert on violations
4. Dashboard key metrics
5. Automate alerting

Distributed Traces

What is a Trace?

Trace: Complete journey of a request

Trace ID: abc123

[Service A] → [Service B] → [Service C]
   50ms         100ms         30ms

Total latency: 180ms

Span

Span: Single operation within a trace

Span ID: span1
Parent: span0
Service: order-service
Operation: create_order
Duration: 50ms
Status: OK

OpenTelemetry Example

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanExporter

# Setup
provider = TracerProvider()
processor = BatchSpanExporter(JaegerExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Create span
with tracer.start_as_current_span('create_order') as span:
    span.set_attribute('order.id', order_id)
    
    # Child span
    with tracer.start_as_current_span('validate_order'):
        validate(order)
    
    # Another child span
    with tracer.start_as_current_span('save_to_db'):
        db.save(order)

Trace Visualization

Trace: abc123 (180ms)

├── order-service.create_order (180ms)
│   ├── validate_order (20ms)
│   ├── inventory-service.check (50ms)
│   │   └── db.query (30ms)
│   ├── payment-service.charge (80ms)
│   │   └── external_api.call (70ms)
│   └── db.insert (20ms)

Sampling

Sampling Strategies:

1. Always: 100% (expensive)
2. Never: 0% (no traces)
3. Rate-based: 10% of requests
4. Adaptive: More on errors
5. Head-based: Decide at start
6. Tail-based: Decide after completion

Best Practices

1. Instrument all services
2. Use correlation IDs
3. Sample strategically
4. Capture errors in spans
5. Visualize trace graphs