Metrics

Counter Metric

Counter is a monotonically increasing metric.

Characteristics

Counter:
- Only increases (or resets to 0)
- Never decreases
- Represents cumulative total

Examples:
- requests_total
- errors_total
- bytes_processed_total

Implementation

from prometheus_client import Counter

# Define counter
REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

# Increment
REQUEST_COUNT.labels(method='GET', endpoint='/api', status='200').inc()

# Increment by value
REQUEST_COUNT.labels(method='POST', endpoint='/api', status='200').inc(5)

Rate Calculation

# Prometheus rate() function
# Rate of requests per second over 5 minutes
rate(http_requests_total[5m])

# Increase over last hour
increase(http_requests_total[1h])

Use Cases

Use Case	Counter Name
Request count	http_requests_total
Error count	errors_total
Messages processed	messages_processed_total
Bytes sent	bytes_sent_total

Best Practices

1. Suffix with _total
2. Use labels for dimensions
3. Use rate() for per-second
4. Use increase() for time window

Gauge Metric

Gauge is a metric that can go up or down.

Characteristics

Gauge:
- Can increase or decrease
- Represents current value
- Point-in-time measurement

Examples:
- temperature
- queue_depth
- cpu_usage
- connections_active

Implementation

from prometheus_client import Gauge

# Define gauge
QUEUE_DEPTH = Gauge('queue_depth', 'Current queue depth')
CPU_USAGE = Gauge('cpu_usage_percent', 'CPU usage', ['service'])

# Set value
QUEUE_DEPTH.set(42)

# Increment/Decrement
QUEUE_DEPTH.inc()  # +1
QUEUE_DEPTH.dec()  # -1
QUEUE_DEPTH.inc(5)  # +5

# With labels
CPU_USAGE.labels(service='api').set(75.5)

Use Cases

Use Case	Gauge Name
Queue depth	queue_depth
CPU usage	cpu_usage_percent
Memory usage	memory_usage_bytes
Active connections	connections_active
Temperature	temperature_celsius

Best Practices

1. Use for current state
2. Don't use for rates (use Counter)
3. Set, don't increment
4. Label for dimensions

Histogram Metric

Histogram measures distribution of values.

Characteristics

Histogram:
- Distribution of values
- Predefined buckets
- Calculates quantiles

Examples:
- request_duration_seconds
- response_size_bytes
- db_query_duration_seconds

Implementation

from prometheus_client import Histogram

# Define histogram
REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'Request duration',
    ['endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]  # Custom buckets
)

# Observe value
REQUEST_DURATION.labels(endpoint='/api').observe(0.25)  # 250ms

# Use as context manager
with REQUEST_DURATION.labels(endpoint='/api').time():
    result = process_request()

Buckets

# Default buckets (seconds)
DEFAULT_BUCKETS = (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)

# HTTP latency buckets
HTTP_BUCKETS = (0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)

# Size buckets
SIZE_BUCKETS = (100, 1000, 10000, 100000, 1000000)

Quantile Queries

# P50 (median)
histogram_quantile(0.5, rate(http_request_duration_seconds_bucket[5m]))

# P95
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# P99
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Use Cases

Use Case	Histogram Name
Request latency	http_request_duration_seconds
Response size	http_response_size_bytes
DB query time	db_query_duration_seconds
Queue wait time	queue_wait_duration_seconds

Percentiles

What are Percentiles?

P50 (Median): 50% of requests faster than this
P95: 95% of requests faster than this
P99: 99% of requests faster than this

Example:
- P50 = 50ms (half requests < 50ms)
- P95 = 200ms (95% requests < 200ms)
- P99 = 500ms (99% requests < 500ms)

Why Percentiles?

Average can be misleading:
- 99 requests: 10ms each
- 1 request: 1000ms
- Average: 20ms (looks good!)
- P99: 1000ms (terrible!)

Percentiles show tail latency

Prometheus Queries

# P50
histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# P95
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# P99
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# P999
histogram_quantile(0.999, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

SLOs with Percentiles

SLOs:
  latency:
    P50: < 50ms
    P95: < 200ms
    P99: < 500ms
  availability: > 99.9%
  error_rate: < 0.1%

Best Practices

1. Track P50, P95, P99
2. Use P99 for SLOs
3. Alert on P99 violations
4. Dashboard all percentiles
5. Compare across services