Rate Limiting and Retry Policy

It is 2:07 PM. Your service retries a failed provider call. So does every other replica. All of them retry at the same instant, all of them fail again, all of them retry in lockstep. The provider, already struggling, now faces a synchronized wall of retries on top of normal traffic. By 2:15 the provider's recovery has failed, your quota is burned for the hour, and the incident review has a new term in it: retry storm.

The mechanics are simple. A provider degrades. Your clients receive 429s and 500s. Your code retries — correctly, because transient failures are real. But every retry is another request to the service that is already failing, and when thousands of clients retry on the same schedule, the retries themselves become the load.

No jitter (thundering herd):
t=0s   ████████████████████  all clients fail together
t=2s   ████████████████████  all clients retry together -> fail
t=4s   ████████████████████  all clients retry together -> fail

With full jitter:
t=0s   ████████████████████  all clients fail together
t=0-2s  █ ██ █ ███ █ ██ █ ██   retries spread across the window
t=2-4s   █ ███ █ ██ █ ██ █     provider recovers under low load

Two disciplines prevent this. Rate limiting caps how fast you send, so your own traffic never becomes the problem. Backoff with jitter staggers retries, so a recovering provider faces a trickle instead of a wall. A third mechanism — the circuit breaker — stops you calling a provider that has already failed. This topic builds all three.

Providers enforce quotas with rate limits: N requests per minute, sometimes per day, sometimes per token of throughput. Exceed the quota and every request fails with 429 until the window resets. The provider's limit is a fact about their infrastructure; your client-side limiter exists so you never discover it by tripping it.

The token bucket is the standard algorithm. A bucket holds tokens at a maximum capacity. Tokens refill at a fixed rate. Each request consumes one token; if the bucket is empty, the request waits or is rejected.

import threading
import time

class TokenBucket:
    def __init__(self, rate: float, capacity: int):
        self.rate = rate            # tokens refilled per second
        self.capacity = capacity   # maximum burst size
        self.tokens = capacity
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def acquire(self) -> bool:
        with self.lock:
            now = time.monotonic()
            elapsed = now - self.updated
            # refill from real elapsed time, not a fixed sleep — this keeps
            # the average rate exact even when requests arrive irregularly
            self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
            self.updated = now
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            return False

bucket = TokenBucket(rate=2.0, capacity=4)   # 2 req/s sustained, burst of 4
[bucket.acquire() for _ in range(6)]
--> [True, True, True, True, False, False]

The burst capacity is the parameter people get wrong. Set it to 1 and you can never handle a user double-clicking. Set it to the provider's per-minute quota and one burst can consume the entire window. A burst of a few seconds' worth of traffic — 4 to 10 — absorbs normal spikes without threatening the quota.

The bucket also composes with queues. When acquire() returns False, you have a choice: reject the request with a 429 of your own, or enqueue it for later. For user-facing traffic, a short queue with a visible 'working on it' state beats an error page — the same trade as queue-based processing, applied at the client.

Rate limiting prevents you from tripping the provider's quota. Backoff handles the failures that happen anyway — and the detail that separates a polite client from a destructive one is the Retry-After header.

When a provider returns 429, it often tells you exactly when to try again: Retry-After: 12 means twelve seconds. Guessing an interval when the provider has volunteered one is how you end up retrying into a penalty window. The hierarchy is: honor Retry-After when present, fall back to exponential backoff with jitter when it is not.

import random
import time

class RateLimitExceeded(Exception):
    pass

def call_with_retry(request: dict, max_attempts: int = 4):
    for attempt in range(max_attempts):
        resp = client.post(URL, json=request)
        if resp.status_code == 429:
            retry_after = resp.headers.get('Retry-After')
            if retry_after:
                time.sleep(float(retry_after))   # provider told us exactly — obey
                continue
            time.sleep(full_jitter(attempt))    # no header: guess, but spread out
            continue
        if resp.status_code >= 500:
            time.sleep(full_jitter(attempt))
            continue
        return resp
    raise RateLimitExceeded()

def full_jitter(attempt: int) -> float:
    ceiling = min(2 ** attempt, 30)    # cap the window so attempt 20 is not a day long
    return random.uniform(0, ceiling)  # randomize the WHOLE sleep, not just a delta

attempt 1 -> 429, Retry-After: 2  -> sleep exactly 2s
attempt 2 -> 500, no header       -> sleep random in [0, 2)
attempt 3 -> 200                  -> return response

The jitter choice matters more than the base. Fixed backoff (sleep(2 ** attempt)) keeps every client synchronized — the same lockstep that caused the storm in the previous chapter. 'Equal jitter' (sleep(2**attempt/2 + random(0, 2**attempt/2)) helps but still correlates. Full jitter — sleeping a random fraction of the whole window — maximally desynchronizes clients while preserving the exponential average, and AWS's research on backoff found it minimizes total completion time under contention.

One more rule: never retry non-transient errors. A 400 (bad request), 401 (bad key), or 404 will fail identically on every attempt. Retry only 429 and 5xx — the failures that can plausibly succeed later.

Rate limiting and backoff assume the provider eventually answers. A circuit breaker assumes it might not — and stops you from finding out the expensive way.

Without a breaker, every request during an outage waits for a timeout or a 429, ties up a worker, and burns quota on retries. With a breaker, consecutive failures trip the circuit open and subsequent requests fail fast — no provider call, no timeout, no retry budget spent. The service sheds load instead of drowning in it.

The half-open state is the clever part. An open circuit that never closes means a provider recovering at minute 31 stays unused forever. The probe tests the water with a single request: success closes the circuit and traffic resumes; failure re-opens it for another cooldown.

import time

class CircuitBreaker:
    def __init__(self, failures_to_open: int = 5, cooldown: float = 30.0):
        self.failures_to_open = failures_to_open
        self.cooldown = cooldown
        self.failures = 0
        self.opened_at = None

    def call(self, fn):
        if self.opened_at:
            if time.monotonic() - self.opened_at < self.cooldown:
                raise ProviderDown()          # fail fast: no timeout, no retry budget
            self.opened_at = None             # cooldown elapsed -> half-open probe
        try:
            result = fn()
            self.failures = 0                 # any success heals the circuit
            return result
        except ProviderError:
            self.failures += 1
            if self.failures >= self.failures_to_open:
                self.opened_at = time.monotonic()
            raise

The breaker composes with everything else in this topic: the token bucket caps your rate, backoff with jitter spaces out retries, and the breaker stops the retries from mattering when the provider is genuinely down. Together they are the client-side half of resilience; the server-side half — what your service does when every provider is down — is model fallback and degradation.