Replication Fundamentals

Your orders database runs on one Postgres primary with two replicas behind it. A user places an order, the API returns 201 Created, and half a second later their order history page — which reads from a replica — shows nothing. The write succeeded; the read lied. That gap is not a bug in your ORM, it is the defining property of single-leader replication: writes funnel to one node and reads fan out to followers, so every follower is a snapshot of the past.

The mechanism is the same in Postgres, MySQL, and MongoDB. The primary appends the change to its write-ahead log (WAL), ships those log records to followers over a replication stream, and each follower replays them in order against its own copy. Followers never take the client write directly; their entire job is to apply the leader's log and serve reads. Because the stream is a sequence, followers hold a prefix of the leader's history — which is exactly why a replica can be behind rather than wrong.

The one decision that shapes everything is when the leader waits for followers before acknowledging:

RPO (recovery point objective) is the number of writes you lose on failover; RTO is how long recovery takes. Fully asynchronous replication gives RPO ≈ 0 only if nothing bad happens — a primary loss can silently discard seconds of writes that clients were told were safe. This is the classic interview question about replication: how much data can you lose, and does the business care? A payments ledger argues for synchronous or semi-sync to one follower; a clickstream table shrugs at a few lost seconds.

Now the read path. Sending all reads to followers raises the stale-read problem above, plus two specific failures:

read-your-own-writes:  user updates profile → write hits primary (t=0)
                       next request lands on replica that applied up to t=-1.4s
                       → user sees their OLD profile and assumes the save failed

monotonic reads:      request 1 → replica A (state at t=10)
                      request 2 → replica B (state at t=8)  → time appears to run backward

Fixes, in order of cost: sticky sessions pin a user to one replica for a short window; session tokens carry the primary's last-seen timestamp so the router skips replicas behind it; routing by causality sends any read following a write to the primary itself. Notice what none of these do — they do not make the cluster consistent, they make your workload's access pattern consistent with the replication lag that exists anyway.

Failover is where the design gets tested. The primary dies; a monitor (Patroni, pg_rewind, cloud health checks) waits a heartbeat timeout, picks a follower, and promotes it. Two hazards follow. First, the promoted replica is the most advanced one, and writes that never reached it are gone — that is the RPO window materializing. Second, the old primary may still be alive, isolated rather than dead, still accepting writes from clients that can reach it. Unless promotion is fenced — the old primary is killed or its writes rejected by a quorum it can no longer reach — you get two primaries diverging, which is split brain. Raft solves this with majority election (Consensus Overview with Raft); many classic setups solved it with a fence file and STONITH (shoot the other node in the head). Either way the rule is: a node that might be a stale leader must not be able to commit writes.

Expect an interview to ask you to draw the write path, name the lag source, and say what breaks during promotion. Draw the log stream, not a box labeled "replication".

A global product has users in the US and in Europe. Replicating every US write to Frankfurt and back makes the European checkout feel like a cross-Atlantic round trip, so the team puts a primary in each region and lets each accept local writes. Availability and latency improve immediately — and so does a problem that did not exist before: two leaders can now accept writes to the same row while they cannot talk to each other.

Multi-leader replication (active/active, or master-master) is chosen for geography, for writes that are naturally partitioned (each office edits its own documents), or for offline clients that sync later. The replication between leaders is no longer a single ordered stream — each leader's log must be shipped to the others, and messages can arrive out of order, duplicated, or after a network heals. When both leaders wrote price = 99 and price = 109 to the same product, no transport-level rule picks a winner. You have moved the problem from replication into conflict resolution.

Resolution happens at two moments, and the distinction matters:

• Write time (immediate): the moment a leader receives a write, it must decide how that write merges with a competing one it has not yet seen. This keeps the system responsive but forces the rule to be simple.
• Read/repair time (deferred): all writes are accepted and replicated; conflicts are detected when replicas merge, and a resolver runs then. Reads may see intermediate states.

The catalog of strategies, with their honest costs:

Say it plainly in an interview: LWW is not a conflict resolution strategy, it is a conflict deletion strategy — it guarantees convergence (all replicas agree) by throwing away a write. For a shopping cart that is sometimes acceptable; for an inventory decrement or a bank balance it is data loss.

The second hazard is subtler than losing a write: a conflict can be cascading. Suppose leadership updates title and body in two operations, and the two regions apply them in different orders. Region A's version says title=X, body=Y, region B's says title=Y, body=X. Resolving at read time by "whichever arrived last" can resurrect an older combination — this is the write-skew / lost-update family showing up in replication, and it is why guarding a read-modify-write with a version column (UPDATE ... WHERE version = 3) still matters even with one leader per region.

Multi-leader designs also break assumptions you forgot you were making: auto-increment primary keys collide (use UUIDs or per-region key ranges), unique constraints can only be enforced where the write landed (deferred to merge time), and triggers that run twice on merged data produce duplicated side effects. Replication topology adds its own vocabulary — star (one hub) concentrates failure, linear/daisy-chain delays propagation at each hop, all-to-all spreads fastest but makes ordering hardest to reason about.

Use multi-leader when you can name the conflicts and tolerate merging them. If you cannot name them, you want one leader per shard and quorum reads — the leaderless design next, which buys tunable durability without pretending conflicts do not exist. When you finish, continue to Partitioning and Sharding to see how the write load gets split across nodes in the first place.

Dynamo, Cassandra, Riak, and Amazon's own DynamoDB-style designs remove the leader entirely: any node can accept a write, because a leader is a coordination bottleneck and a failover waiting to happen. The replacement is arithmetic. With n copies of each key, a write succeeds after w replicas acknowledge and a read succeeds after returning r replicas. The one rule that makes the whole thing work:

w + r > n     (the read set and the write set MUST overlap)

n = 3, w = 2, r = 2   → 2 + 2 > 3 ✓  any read sees at least one fresh copy
n = 3, w = 1, r = 1   → 1 + 1 ≤ 3 ✗  read can hit a replica that never saw the write
n = 3, w = 3, r = 1   → full write durability, cheap reads
n = 3, w = 1, r = 3   → fast writes, expensive reads

That overlap inequality is the interview answer to "how does a leaderless store guarantee consistency?" — the read quorum and write quorum share at least one node, so at least one returned value is current. It also shows you the dial: raising w makes writes durable but slower (more acks, more round trips), raising r makes reads fresh but heavier. Typical production settings are w=quorum, r=quorum (2 of 3) for balanced workloads, w=1, r=quorum for ingest-heavy telemetry, and w=quorum, r=1 for read-heavy caching where a slightly stale value is fine.

Three implementation details separate the theory from a running cluster:

Sloppy quorums and hinted handoff. The strict rule says the w acks come from the nodes that own the key per the hash ring. In practice a node may be down, so Dynamo-style systems write to any w healthy nodes and stash a hint — "this write belongs to node X" — delivering it when X returns. This raises availability (you never refuse a write just because the owner is down) at the cost of temporarily violating the placement rule; read repair and hinted handoff restore it.

Read repair and anti-entropy. A read that returns r copies compares versions; if one is stale, the reader pushes the newest value back. That fixes divergences a client happens to notice. Background anti-entropy (Merkle trees comparing ranges) catches divergences nobody read — without it, a node that was offline for a day stays quietly wrong forever.

Versioning, not timestamps alone. Each write gets a version — a Lamport-style counter, a hybrid timestamp, or a vector — and merges pick by version. This is where the causal work you met in Logical Clocks and Ordering pays off: a pure wall clock lets clock skew erase writes, so Cassandra added last-write-wins with bounded clock drift and HLC-style ordering elsewhere.

What quorum does not give you: it does not make a read-following-a-write safe when w=1, r=1, it does not stop two concurrent increments from losing one (you need a compare-and-set on the version, or a CRDT counter), and it does not survive n failures worse than n - w tolerated write failures. Quorums are tunable eventual consistency with a mathematical floor, not a substitute for consensus — which is why systems that need a single ordered truth still elect a leader (Consensus Overview with Raft).

A strong interview answer states the inequality, gives the overlap argument in one sentence, and then immediately names what it does not buy you. Follow the roadmap onward to Partitioning and Sharding, where the question becomes how many nodes hold each key at all.