Leader Election¶
Why Does Leader Election Exist?¶
Many distributed systems need exactly one node acting as the coordinator or primary at any given moment — a single node that accepts writes, or that makes a specific class of decision, while every other node defers to it. The Replication chapter already introduced why this matters: without exactly one primary, you risk split-brain — two nodes both believing they're in charge, both accepting writes independently, producing data that silently diverges with no way to reconcile it cleanly afterward.
Leader election is the concrete, practical application of Consensus to solve exactly this: getting a group of nodes to agree on which single one of them is currently "the leader," in a way that's safe even when nodes crash, restart, or become temporarily unreachable.
5-node cluster, no leader yet
↓
Nodes propose themselves, request votes from each other
↓
Whoever gets a MAJORITY of votes becomes leader
↓
Every other node defers to the elected leader until it fails
or becomes unreachable, triggering a new election
The Basic Mechanism¶
1. A node notices there's no current leader (or the current leader
has stopped sending heartbeats) and becomes a CANDIDATE
2. It requests votes from every other node
3. Each node votes for at most ONE candidate per election round
4. The candidate that receives a MAJORITY of votes becomes leader
5. The leader sends periodic HEARTBEATS to prove it's still alive
6. If heartbeats stop arriving (leader crashed, or got partitioned
away), nodes time out and trigger a NEW election
This is Raft's leader election mechanism specifically, described in the Consensus chapter as one of Raft's three decomposed sub-problems — leader election is what that sub-problem looks like fully worked out on its own.
Terms/Epochs — The Detail That Actually Prevents Split-Brain¶
This is the single most important mechanical detail in this entire chapter, and the direct, concrete answer to the split-brain problem the Replication chapter left open.
Every election increments a TERM (or epoch) number:
Term 1: Node A elected leader
Term 2: (Node A crashes, new election) Node B elected leader
Term 3: (Node B crashes, new election) Node C elected leader
Node A (the OLD leader from Term 1) was actually just network-partitioned,
not truly crashed — it comes back online, still believing it's the
current leader, and tries to accept a write.
↓
Every other node responds: "You're from Term 1. We're currently on
Term 3. Your leadership is STALE — rejected."
This is fencing, and it's what actually closes the split-brain gap — not the election itself, but every subsequent interaction being tagged with a term number that lets every node (and every client) recognize and reject a "zombie" leader that thinks it's still in charge but has actually been superseded. Without this, an old leader recovering from a partition could genuinely still act as if nothing happened, and the exact split-brain scenario from the Replication chapter recurs.
Randomized Election Timeouts — Avoiding Repeated Split Votes¶
A subtle, practical detail worth knowing: if every node used the exact same fixed timeout before starting a new election, many nodes could become candidates simultaneously, splitting the vote so no single candidate gets a majority — forcing a repeated, wasted election cycle.
Fixed timeout for everyone: all nodes notice the leader is gone
at roughly the SAME moment → all
become candidates at once → votes
split across many candidates →
NO majority → election fails → retry
Randomized timeout: each node waits a slightly different,
randomly-chosen amount of time before
becoming a candidate → whichever node's
timer fires FIRST usually gets to request
votes before others even become candidates
→ far fewer split votes, faster convergence
Raft specifically uses randomized election timeouts for exactly this reason — a small, deliberate detail that meaningfully reduces how often an election has to be retried before successfully producing a single winner.
What Happens During a Partition, Mid-Election¶
5-node cluster splits into a group of 3 and a group of 2
↓
Group of 3 (has a majority): CAN successfully elect a leader
and continue making progress
Group of 2 (no majority): CANNOT elect a leader — any candidate
on this side can gather at most 2 votes,
never a majority of 5 (which requires 3)
This is the exact same CP-leaning behavior described in the Consensus and CAP Theorem chapters, made concrete here — the minority side deliberately makes no progress at all rather than risk electing its own leader independently, which is precisely what would recreate split-brain.
Real-World Examples¶
Kubernetes Control Plane¶
Multiple replicas of the controller-manager and scheduler run simultaneously for high availability, but only one is actually "active" at a time — leader election among them (coordinated via etcd's own Raft-based consensus) ensures exactly one replica is making scheduling decisions at any moment, with automatic failover to a standby replica if the active one fails.
Kafka's Controller Broker¶
Historically elected via Zookeeper, and in modern Kafka (KRaft mode) elected via Kafka's own built-in Raft implementation — the controller broker is responsible for cluster-wide metadata decisions (which broker is the leader for which partition), and needs exactly one active instance for the same reasons any leader-based system does.
Redis Sentinel¶
A practical, real-world instance of leader election applied to failover — Sentinel nodes monitor a Redis primary, and upon detecting it's unreachable, use a voting mechanism among themselves (requiring a majority/quorum of Sentinels to agree) before promoting a replica to the new primary, specifically to avoid a false positive (a single Sentinel's network hiccup) triggering an unnecessary failover.
Common Interview Questions¶
Why do distributed systems need leader election at all?¶
To guarantee exactly one node is accepting writes or making a specific class of decision at any given time — without this, multiple nodes could independently accept conflicting writes or make conflicting decisions simultaneously, producing the split-brain problem covered in the Replication chapter.
How does leader election actually prevent split-brain?¶
Not through the election mechanism alone, but through term/epoch numbers attached to every subsequent interaction — an old leader that was merely partitioned away (not truly dead) and comes back online is recognized as stale by every other node, since it's still operating under an outdated term number, and its attempts to act as leader are rejected. This fencing mechanism is what actually closes the gap, not the vote itself.
Why does Raft use randomized election timeouts instead of a fixed one?¶
To reduce the chance of multiple nodes becoming candidates at exactly the same moment and splitting the vote so no one achieves a majority. A randomized timeout means one node's timer typically fires first, letting it request votes before other nodes even become candidates — meaningfully reducing how often an election needs to be retried.
What happens if a network partition occurs during an election?¶
Only the side of the partition with a majority of the original node count can successfully elect a leader and continue making progress; the minority side cannot gather enough votes for any candidate to reach a majority, and deliberately makes no progress at all — the same CP-leaning trade-off described in the CAP Theorem and Consensus chapters, applied specifically to the election process itself.
Is leader election alone sufficient to guarantee correctness, or is something else needed?¶
Leader election alone isn't sufficient — it needs to be paired with fencing (term/epoch numbers that let every node reject a stale, superseded leader) to actually prevent split-brain. A system that elects leaders but doesn't tag interactions with a term number is still vulnerable to a recovering "zombie" leader believing it's still in charge.
Common Mistakes¶
❌ Assuming a successful leader election alone guarantees no split-brain, without fencing.
An old leader recovering from a temporary partition, with no term/epoch mechanism to reject it, can still believe it's in charge and act accordingly — the election needs to be paired with fencing to actually close this gap.
❌ Using fixed, non-randomized election timeouts.
Increases the likelihood of repeated split votes, since many nodes are likely to notice the leader is gone and become candidates at nearly the same moment.
❌ Assuming a leader can still be elected during a partition on the side lacking a majority.
This is a deliberate, safety-preserving impossibility, not a bug — proceeding without a majority is exactly what would risk two independently-elected leaders and recreate split-brain.
Key Takeaways¶
- Leader election is the practical, concrete application of consensus to guarantee exactly one node acts as coordinator or primary at a time — directly solving the split-brain risk introduced in the Replication chapter.
- Term/epoch numbers, attached to every subsequent interaction after an election, are the actual mechanism (fencing) that prevents a recovering, previously-partitioned leader from acting as if it's still in charge — the election itself isn't sufficient on its own.
- Randomized election timeouts (as used in Raft) reduce the chance of repeated split votes by making it more likely one node's timer fires first, before other nodes become competing candidates.
- During a partition, only the side with a majority of nodes can successfully elect a leader — the minority side deliberately makes no progress, the same CP-leaning trade-off described in Consensus and CAP Theorem, applied specifically to elections.
- Real systems (Kubernetes, Kafka, Redis Sentinel) all rely on some form of majority-based leader election specifically to provide automatic, safe failover without risking two active leaders existing at once.