Skip to main content
PreparationZone Logo
PrepZone

Leader Election and Failover

One writer at a time, clean failover, and fencing stale leaders — ShardPay's shard primaries.

Pass checkpoint below

Why this matters

  • Two leaders writing the same shard causes irreconcilable ledger state — worse than temporary unavailability.
  • Failover speed vs safety is the core trade-off; aggressive timeouts cause flapping, conservative timeouts extend outage.
  • ShardPay uses Raft-based election per ledger shard and for control-plane metadata.
  • Interviewers ask how you prevent split-brain and what happens to in-flight writes during failover.
FollowerElection timeout
CandidateRequests votes
LeaderAccepts client writes
Followers timeout, become candidates, request votes. Majority wins.

Safe failover

When ShardPay's shard leader dies, followers increment their election timer. After a randomized timeout (150–300ms), a follower campaigns for votes. A candidate needs a majority quorum to become leader, appends a new term, and rejects writes from any node with a stale term. The old leader, if it recovers from a network glitch, cannot commit — its term is lower and clients have moved on.

Election mechanics

  • Election timeout — Follower waits random interval before starting election; reduces split-vote probability when multiple nodes timeout together. ShardPay tunes base timeout to 200ms with ±50ms jitter on ledger shards in one region (RTT < 2ms).
  • Quorum (majority vote) — Candidate must receive votes from more than half of cluster members. ShardPay's 5-node shard requires 3 votes — a partitioned minority cannot elect a leader.
  • Term / epoch — Monotonic generation number; higher term wins. Every ShardPay Raft RPC carries term; stale-term responses trigger client redirect to current leader.
  • Log completeness check — Voters reject candidates whose log is behind. ShardPay ensures the new leader has all committed entries before serving writes — no lost transfers on failover.
  • Split-brain — Two nodes both believing they are leader. Prevented by quorum requirement + fencing: storage rejects writes without current-term lease.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Election lifecycle
  • Tuning and operations

Test yourself

Answer these before moving on — recall is what makes it stick.