Logging

Structured Logging

Structured logging outputs logs in a machine-parseable format (JSON).

Why Structured?

Unstructured:
2024-01-15 10:00:00 INFO Order 123 created for user 456

Structured:
{
  "timestamp": "2024-01-15T10:00:00Z",
  "level": "INFO",
  "message": "Order created",
  "order_id": "123",
  "user_id": "456",
  "service": "order-service"
}

Structured: Searchable, analyzable, aggregatable

Implementation

import json
from datetime import datetime

class StructuredLogger:
    def __init__(self, service_name):
        self.service = service_name
    
    def _log(self, level, message, **kwargs):
        entry = {
            'timestamp': datetime.utcnow().isoformat() + 'Z',
            'level': level,
            'service': self.service,
            'message': message,
            **kwargs
        }
        print(json.dumps(entry))
    
    def info(self, message, **kwargs):
        self._log('INFO', message, **kwargs)
    
    def error(self, message, error=None, **kwargs):
        if error:
            kwargs['error'] = str(error)
            kwargs['stack_trace'] = traceback.format_exc()
        self._log('ERROR', message, **kwargs)

logger = StructuredLogger('payment-service')
logger.info('Payment processed', payment_id='789', amount=99.99)

Best Practices

1. Always use JSON format
2. Include timestamp (UTC)
3. Include service name
4. Include correlation ID
5. Don't log sensitive data (PII, passwords)

Log Levels

Level Definitions

Level	When to Use	Example
DEBUG	Detailed debugging info	Variable values, function entry/exit
INFO	Normal operations	Request processed, order created
WARN	Unexpected but handled	Retry attempt, fallback used
ERROR	Errors requiring attention	Failed to connect, exception occurred
FATAL	System-threatening	Out of memory, unrecoverable error

Level Configuration

import logging

# Set level based on environment
log_level = os.getenv('LOG_LEVEL', 'INFO')
logging.basicConfig(
    level=getattr(logging, log_level),
    format='%(asctime)s %(levelname)s %(message)s'
)

# Production: INFO or WARN
# Development: DEBUG
# Testing: WARNING

Level Usage

def process_order(order):
    logger.debug(f'Processing order: {order.id}')  # Detailed
    
    try:
        validate(order)
        logger.info(f'Order validated: {order.id}')  # Normal
        
        if order.amount > 10000:
            logger.warn(f'Large order amount: {order.amount}')  # Unusual
        
        save(order)
    except ValidationError as e:
        logger.error(f'Validation failed: {e}', order_id=order.id)  # Error
        raise
    except Exception as e:
        logger.fatal(f'Unexpected error: {e}')  # Critical
        raise

Best Practices

1. DEBUG: Detailed, verbose, development only
2. INFO: Normal operations, business events
3. WARN: Handled anomalies, worth monitoring
4. ERROR: Failures requiring attention
5. FATAL: System-threatening, immediate action

Centralized Logging

Why Centralize?

Distributed System:
- Multiple services
- Multiple servers
- Multiple logs

Problem: Can't search across logs!

Solution: Centralized logging
- All logs → Central store
- Search across all services
- Correlate events

Architecture

Service 1 → [Agent] ─┐
Service 2 → [Agent] ─┼→ [Log Aggregator] → [Storage] → [Search UI]
Service 3 → [Agent] ─┘

Agents: Filebeat, Fluentd, Fluent Bit
Aggregator: Logstash, Fluentd
Storage: Elasticsearch, Loki
UI: Kibana, Grafana

ELK Stack

Elasticsearch: Storage and search
Logstash: Processing and enrichment
Kibana: Visualization and search

Flow:
Apps → Filebeat → Logstash → Elasticsearch → Kibana

Loki (Lightweight)

# Loki + Promtail config
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ['localhost']
        labels:
          job: myapp
          host: server1

Log Enrichment

class LogEnricher:
    def __init__(self):
        self.metadata = {
            'hostname': socket.gethostname(),
            'service': os.getenv('SERVICE_NAME'),
            'environment': os.getenv('ENVIRONMENT')
        }
    
    def enrich(self, log_entry):
        return {
            **self.metadata,
            **log_entry,
            'trace_id': get_trace_id(),
            'correlation_id': get_correlation_id()
        }

Best Practices

1. Use structured logging
2. Centralize all logs
3. Enrich with context
4. Set retention policies
5. Index important fields
6. Monitor log volumes