Caching Strategies

A recent autoscaling review from Load Balancing and Autoscaling told you to add instances until latency recovers — but your database was the bottleneck all along, and instances do not fix a database. Your catalog endpoint hits PostgreSQL on every request. A Redis cache sits one network hop away and nobody agrees on who is responsible for filling it. One engineer wraps every repository call in get → miss → query → set; another passes a loader function into the cache library; a third writes to the cache first and lets it push to the database. All three are named patterns, and they fail differently.

The patterns differ on two axes: who owns the miss logic, and where the write goes first.

Cache-aside is the industry default because it keeps the cache optional. The application is the only component that knows caching exists:

def get_product(pid: int) -> dict:
    raw = redis.get(f"product:{pid}")
    if raw is not None:
        return json.loads(raw)                 # hit: ~0.5 ms, no DB involved
    product = db.fetch("SELECT * FROM products WHERE id = %s", pid)
    if product:                                # optional: skip caching nulls,
        redis.setex(f"product:{pid}", 300, json.dumps(product))   # or negative-cache briefly
    return product

The costs: the logic is duplicated at every call site (and drifts — one repo forgets setex), the first request after every miss pays the full database latency, and nothing stops a thousand concurrent misses from all querying at once. The win: when Redis is unreachable, reads still work.

Read-through moves that logic inside the cache layer. The application calls cache.get(key) and the cache's configured loader fetches on miss — Caffeine's LoadingCache, or a cache service wired with a database callback. Two properties make this the better default where available: there is exactly one implementation of the miss path, and the cache library provides request coalescing — while one caller populates a key, every other caller for that key waits and then reads the fresh value instead of issuing its own query. Cache-aside gives you no such guarantee until you build it yourself.

Write-through makes writes flow cache → database synchronously. Reads immediately after the write find the new value, which is the property you cannot get from delete-on-write. The prices: write latency becomes cache + database (say 1 + 12 ms instead of 12 ms — usually acceptable), the database must absorb every write the cache accepts (a cache does not reduce write load; it can only absorb it if you batch — see write-behind), and the cache is now on the write path, so a cache outage can block writes unless you plan a bypass.

Write-behind is write-through with a buffer: acknowledge after the cache, flush to the database in batches. It is the only pattern that reduces database write load — batching 5,000 single-row transactions into 50 batched ones eliminates thousands of WAL fsyncs — and the only one where a cache crash loses acknowledged data.

Interview shorthand: cache-aside = application owns misses, read-through = cache owns misses, write-through = synchronous dual-write for read-after-write freshness, write-behind = buffered writes for throughput at the risk of loss. Choosing is mostly about writes, which is exactly where the interesting failures live — the next chapter.

A user renames their profile. Your code updates PostgreSQL and then SETs the cache with the new value. Two seconds later the profile shows the old name for a minute. The write "succeeded" everywhere and users still see stale data — because the write path, not the read path, is where cache consistency is decided.

Why delete, not update. The default cache-aside write is: update the database, then DEL the key. Updating the key instead invites a lost-update race — two concurrent writers, A and B, each write their own value into the cache, and the cache can end up holding A's version while the database holds B's:

DB:      W_A commits (name=Ada) ... W_B commits (name=Grace)      ← DB is correct
cache:   SET name=Ada  (in flight, delayed 200 ms)
         SET name=Grace
         SET name=Ada  ← stale value now cached, DB says Grace

Deletion removes the risk of writing a stale value — the next read reloads from the database — at the cost of one miss. Order matters: commit the database write first, delete second. Delete before commit and a racing reader repopulates the pre-write value, which then survives until TTL. If the delete can fail (network partition), retry it or fall back to a short TTL as the safety net.

Write-around (write-no-allocate). For write-heavy data that will not be read soon — audit rows, import batches — writing through the cache only to evict it later pollutes your cache with cold keys and pushes hot ones out under LRU pressure. Write directly to the database and let the next read allocate the cache entry. The trade: nothing is warm immediately after a write, so read-after-write workloads pay a miss they did not need.

Write-through vs write-behind, with numbers. Suppose peak write load is 5,000 rows/second and each database commit costs an fsync (1–10 ms on typical storage):

write-through / direct:   5,000 transactions/s → 5,000 fsyncs/s → the WAL is the bottleneck
write-behind (flush every 1 s, group rows):
                          5,000 rows → ~1 transaction with 5,000 rows via COPY/batch
                          fsyncs drop by 3 orders of magnitude; write latency becomes
                          "acknowledged in cache" (~1 ms) instead of "durable on disk"

That is the entire appeal of write-behind: it converts random single-row commits into sequential bulk work. The cost is a durability window — if the node holding the buffer dies before flush, those acknowledged writes are gone — plus reordering hazards (two batches for the same row must flush in order). Accept it for session state, view counters, telemetry, and metrics where losing the last second is fine; never for balances, inventory reservations, or anything an auditor will ask about.

The honest interview answer to "how do you keep cache and database consistent?" is layered: delete-after-commit for the fast path, a bounded TTL as the always-correct backstop, and — when "the user just changed it and must see it" is a hard requirement — either write-through or a per-entity version check on the next read. Deleting keys naively also breaks down when many keys depend on one row; at that point the problem stops being about write paths and becomes invalidation proper, which we return to in Cache Invalidation Trade-offs. First, though, the failure mode that happens without any writes at all.

No incident, no deploy, no traffic spike — just a clock. At 14:03:00 the TTL on your hottest cache key expires, 12,000 requests per second are all looking for that one key, and every single one of them misses and queries PostgreSQL. Connection pool saturates in 200 ms, requests queue, latencies cross the timeout line, retries arrive, and the database falls over under a load your cache was built to prevent. This is a cache stampede (a.k.a. thundering herd on expiry), and the related variant — many clients hitting a slow dependency at once — is the same physics with a different trigger.

Defenses, in the order you should apply them:

1. Request coalescing (single flight). Only one in-flight load per key; everyone else waits for it. NGINX ships it:

location /api/ {
    proxy_cache         api_cache;
    proxy_cache_lock    on;             # only one request populates a given key
    proxy_cache_lock_timeout 5s;        # waiters time out → they may query origin
    proxy_cache_use_stale updating timeout error http_500 http_503;
}

2. A distributed lock with a lease. SET lock:product:9 1 NX PX 5000 — the winner recomputes; losers serve stale or wait, and the 5 s expiry guarantees a crashed holder cannot wedge the key forever. Always pair the lock with "serve stale if not acquired"; blocking 12,000 requests on a lock is a latency incident of its own.

3. Jittered TTLs. If a thousand keys were all written during a deploy, they all expire in the same second. Spread it:

expire_at = now + base_ttl + random(0, 0.2 × base_ttl)
# base TTL 300 s ± 60 s → expirations smear across 2 minutes instead of spiking in one

4. Probabilistic early refresh (XFETCH). Before expiry, refresh with probability proportional to how old the entry is: refresh if rand() < C × (age / ttl) with C ≈ 1. Expired-in-practice keys get rebuilt slightly early, one at a time, at random — no synchronized wake-up. It needs an internal timestamp (or a logical clock) inside the cached value.

5. stale-while-revalidate. Serve the stale response and refresh in the background (Cache-Control: max-age=60, stale-while-revalidate=300). Users never wait for computation; the window of staleness is explicit and bounded.

Then respect the 90/10 rule. Roughly 90% of traffic lands on 10% of keys — your front page, the app's splash config, one celebrity's profile. That concentration is not a nuisance; it is the design input:

• Measure hit ratio per key, not just aggregate. An 80% overall ratio can hide that your top 10 keys are being evicted.
• Give hot keys long TTLs and admission control: TinyLFU/Window-TinyLFU (Caffeine's default) admits a new entry only if it is predicted to be more frequent than the eviction candidate, so a burst of one-hit wonders cannot evict your 90% keys.
• If hot keys still overload one cache shard, replicate them: cache key plus key:2, key:3 on different nodes and pick randomly on read — a classic fix for a single hot partition.
• Cache nothing that is read once. Cold keys cost memory and load-creation for zero hit ratio; negative caching ("product does not exist") is worth it only with a short TTL.

The arithmetic that makes this concrete: at 10,000 requests/second, a 95% hit ratio leaves 500 req/s for the database; a stampede that drags it to 80% sends 2,000 req/s — four times the origin load from a 15-point hit-ratio drop. That is why coalescing plus jitter are not micro-optimizations. They are the difference between a cache that protects your database and one that schedules its outages for you.

So far every cache here lives inside your data center. The next topic moves the same questions — keys, TTLs, coalescing — to a global network: CDN and Edge Caching.