Consensus Overview with Raft

Three servers replicate a key-value store. The client writes x = 5 to server A while a concurrent write x = 7 reaches server B. Network delays reorder the replication traffic, and when the dust settles A believes the key is 5 and B believes it is 7 — reads from different replicas now return different answers, and no amount of timestamping says which write "really" happened. This is the consensus problem: given processes that may crash and a network that may delay or drop messages, get the replicas to agree on one value (or one sequence of values) — and keep agreeing.

The specification splits in two, and every serious discussion uses these names:

Safety is never traded: a system that is consistently wrong can be repaired by replaying; a system that is inconsistently correct has lost a fact that no one can reconstruct. Liveness is traded all the time — during a partition, choosing to refuse writes is the correct trade (the PACELC/CAP discussions you will meet return exactly here).

Two theoretical results shape the design space. FLP (1985) proves that in a fully asynchronous system, even one crash-prone process can prevent consensus from terminating — so protocols must assume some timing structure (message timeouts are an assumption, not an implementation detail). Impossibility without a majority is the second: if the network splits a set of replicas so that no majority side can talk to anyone, two sides could each elect a leader and each commit different values — so every correct protocol requires a majority quorum to act at all. Quorums are not a tuning parameter; they are how safety survives partitions.

Raft's contribution is not a new impossibility result but an unusually understandable structure. Where Paxos is a family of arguments about single entries, Raft is deliberately engineered for comprehension — a design goal you can cite in an interview — and its decomposition is the memory anchor:

leader election   — one writer at a time, numbered by monotonic terms
log replication   — leader appends, followers mirror; the leader alone proposes
safety argument   — majorities overlap, so 'committed' entries are on every future
                    leader's log; elections cannot install an older log as newer

Each of the next two chapters takes one piece. Before that, the framing that keeps Raft in perspective: consensus is the primitive, not the product. Real systems use a Raft-replicated log as the substrate — etcd stores Kubernetes' desired state in it, ZooKeeper drives coordination, Consul holds service registrations — and build sessions, leases, watches, and multi-key transactions on top. When an interviewer asks "what does Raft give you?", the precise answer is: an append-only log that all correct replicas agree on, with the properties that make it usable as a foundation. Knowing what sits above the log is as much a part of the answer as the election rules — and knowing what Raft refuses to promise (we will come back to this) separates users from readers.

Split brain is the nightmare: two leaders, each certain of its legitimacy, each committing different values. Raft prevents it with an arithmetic trick and a numbering scheme — no global clock required.

Elections start randomly, not synchronously. Each follower runs an election timeout (typically 150–300 ms, randomized per node so they do not collide). If no heartbeat from a leader arrives before the timeout, the follower increments the term, becomes a candidate, votes for itself, and requests votes from peers. A voter grants at most one vote per term (it remembers the candidate it voted for), and grants it only if the candidate's log is at least as up to date as its own — the rule that stops a stale replica from winning with an old log.

The three mechanisms that collapse split-brain risk:

1. terms are monotonically increasing integers stamped on EVERY RPC — any message
   from an older term is ignored, so a delayed packet cannot resurrect yesterday's leader
2. at most one leader per term — a candidate needs a MAJORITY of votes; two majority
   sets over n nodes must share at least one member, and that member votes only once
   per term ⇒ at most one candidate can reach n/2+1 votes in a given term
3. leaders learn they've been deposed by rejection: a reply carrying a higher term
   makes the old leader step down immediately, even if it never saw the new one

Point 2 deserves its arithmetic because it is the safety argument: with 5 nodes, a leader needs 3 votes. Two disjoint sets of 3 cannot exist inside 5 nodes — they overlap — so if any node has already voted in term 4, no rival can collect a second majority of term 4. A split vote (both candidates get 2) simply produces no leader; the randomized timeouts re-trigger, a new term begins, and liveness resumes. Split-brain risk is controlled because only one leader can win a given term's election — and stale leaders are controlled because every RPC carries its term.

The failure story that makes this concrete: a leader in region A is partitioned from the majority. It still believes it is leader — Raft leaders do not voluntarily resign from silence. Clients on its side may accept heartbeats, but the moment it tries to replicate an entry it cannot gather a majority ack, so it cannot commit anything. Meanwhile the majority side times out, holds a new election in a higher term, elects a new leader, and serves reads/writes normally. When the partition heals, the old leader receives a higher-term reply, steps down, and Raft's log-repair rules discard its uncommitted tail. Two leaders briefly existed; only one could ever commit — which is the property that actually matters.

Practical consequences worth citing: election timeouts must be substantially larger than heartbeat intervals (10× is typical) or heartbeats become noise; a minority partition is a read-only-or-dead region by design — clients must be routed or accept stale reads explicitly; and under load, followers that fall behind on disk fsync can timeout spuriously (the well-known Raft liveness wobble in etcd and similar systems), which is why commit latency is dominated by fsync discipline and why production deployments tune election timeout against measured tail disk latency. The election chapter is where Raft's cost becomes visible: safety came from quorums, and quorums mean a partitioned minority cannot serve — a fact every availability design must confront head-on.

Election won; now the log. A client sends SET x=5 to the leader. The leader appends a new log entry — (term 7, index 104, command) — to its private log and fires AppendEntries RPCs to followers in parallel. Each follower appends the entry only after checking consistency: the RPC carries the previous entry's (term, index), and a follower whose log does not match at that position rejects — which triggers the leader to walk its nextIndex backward until the logs realign, overwriting the follower's divergent tail. That rejection-and-truncate loop is how Raft heals a follower that was partitioned while an old leader wrote to it.

A majority quorum is required before entries are considered committed durably. The rule, stated exactly: an entry from the current term is committed when the leader has replicated it to a majority — measured by matchIndex, each follower's highest known-matching index, taking the median of {self + followers} — and the leader applies it to its state machine. Followers learn commit status piggybacked on the next heartbeat/AppendEntries (leaderCommit field), apply the entry, and respond.

t=7  client → leader: SET x=5
     leader appends (7,104)          ← not committed yet
     AppendEntries → f1, f2, f3, f4
     f1 ack ✓  f2 ack ✓              ← majority of 5 = 3 including leader ⇒ COMMITTED
     f3, f4 lag; healed by next round
     leader applies x=5 to state machine, returns success to client
     heartbeat carries leaderCommit=104 → followers apply

Follow the crash points, because they are the interview material. Leader dies after appending but before acks: entry is uncommitted — the next leader retries or truncates it; the client, which never got a success, may safely retry (commands carry idempotency context). Leader dies after majority ack but before telling the client: the entry is committed — the new leader has it (majority overlap guarantees any future leader holds every committed entry) and completes the application; the client's timeout resolves via read-your-writes or a status query. Follower crashes mid-append: its persisted tail is either matching or truncated on recovery; Raft's per-index term comparison guarantees the state machine applies the same prefix on every node — replicas may differ in their uncommitted tails, never in what has been applied.

What you get, and what you deliberately do not:

The architectural summary to carry out of this topic: Raft buys single-leader agreement through monotonic terms plus overlapping-majority quorums, commits only what a majority has durably stored, and pays with minority unavailability and leader-bound write throughput. When a design brief says "we'll use Raft/etcd for coordination," the checks are: is the write volume within a single fsync-bound leader? What happens to your availability during a minority partition? And what application-level semantics (sessions, leases, watches) will you build on the raw log? Answer those three and you are using consensus as an engineer rather than as a name.