Why this matters
- Two leaders writing the same shard causes irreconcilable ledger state — worse than temporary unavailability.
- Failover speed vs safety is the core trade-off; aggressive timeouts cause flapping, conservative timeouts extend outage.
- ShardPay uses Raft-based election per ledger shard and for control-plane metadata.
- Interviewers ask how you prevent split-brain and what happens to in-flight writes during failover.
FollowerElection timeout
CandidateRequests votes
LeaderAccepts client writes
Safe failover
When ShardPay's shard leader dies, followers increment their election timer. After a randomized timeout (150–300ms), a follower campaigns for votes. A candidate needs a majority quorum to become leader, appends a new term, and rejects writes from any node with a stale term. The old leader, if it recovers from a network glitch, cannot commit — its term is lower and clients have moved on.
Election mechanics
- Election timeout — Follower waits random interval before starting election; reduces split-vote probability when multiple nodes timeout together. ShardPay tunes base timeout to 200ms with ±50ms jitter on ledger shards in one region (RTT < 2ms).
- Quorum (majority vote) — Candidate must receive votes from more than half of cluster members. ShardPay's 5-node shard requires 3 votes — a partitioned minority cannot elect a leader.
- Term / epoch — Monotonic generation number; higher term wins. Every ShardPay Raft RPC carries
term; stale-term responses trigger client redirect to current leader. - Log completeness check — Voters reject candidates whose log is behind. ShardPay ensures the new leader has all committed entries before serving writes — no lost transfers on failover.
- Split-brain — Two nodes both believing they are leader. Prevented by quorum requirement + fencing: storage rejects writes without current-term lease.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Election lifecycle
- Tuning and operations
Test yourself
Answer these before moving on — recall is what makes it stick.