Why this matters
- This article consolidates the Distributed Systems track into an interview-ready framework.
- Interviewers at senior/staff level rarely ask trivia — they present ambiguous problems and evaluate structured thinking.
- ShardPay examples from modules 1–12 provide concrete anchors for every abstract trade-off.
- Revisit this article after completing all modules for spaced recall before interview loops.
The four-step framework
Every distributed systems design answer at ShardPay interview bar follows four steps. Skip any step and the answer feels junior.
Step 1: Clarify requirements
Before drawing boxes, state assumptions:
- Scale: 1M merchants, 3M transfers/day, peak 500/sec
- Consistency: ledger CP (no double-spend); activity feed AP
- Latency: p99 transfer < 200ms; cross-shard acceptable at 400ms
- Durability: no acknowledged transfer lost; reconciliation within 24h
- Availability: 99.95% transfer SLO; brief read staleness OK for dashboards
Ask the interviewer which dimension matters most if time is short. "If I can only optimize one property, should it be correctness or latency?"
Step 2: Draw components and data flow
Sketch: API gateway → fraud → orchestrator → shard A debit → shard B credit → outbox → settlement consumer. Label sync vs async, CP vs AP per component.
Name ShardPay's actual choices: orchestrated saga (not 2PC), consistent hash sharding, cell-based routing, Kafka at-least-once with idempotent consumers.
Step 3: Name failure modes
For each arrow in the diagram, ask "what if this fails?"
| Failure | Impact | ShardPay mitigation |
|---|---|---|
| Network partition between shards | Saga stuck in DEBITED | Step timeout + compensation |
| Coordinator crash | Saga pause, not duplicate | Durable saga log + leader election |
| Duplicate Kafka event | Double credit risk | Idempotency key on consumer |
| Hot shard | p99 latency spike | Consistent hash + sub-shards |
| AZ outage | 1/3 replica loss | Quorum 2/3 writes continue |
| Bad deploy | Regression in transfer path | Cell-scoped canary + rollback |
Network partition between shards
ImpactSaga stuck in DEBITEDShardPay mitigationStep timeout + compensationCoordinator crash
ImpactSaga pause, not duplicateShardPay mitigationDurable saga log + leader electionDuplicate Kafka event
ImpactDouble credit riskShardPay mitigationIdempotency key on consumerHot shard
Impactp99 latency spikeShardPay mitigationConsistent hash + sub-shardsAZ outage
Impact1/3 replica lossShardPay mitigationQuorum 2/3 writes continueBad deploy
ImpactRegression in transfer pathShardPay mitigationCell-scoped canary + rollback
Common failure modes and mitigations
Step 4: Propose mitigations and observability
Mitigations without observability are unverifiable. Every ShardPay answer ends with: "I'd know this works because..."
- Transfer SLI at gateway with 99.95% SLO and error budget alerts
- Distributed traces with transferId baggage across all saga steps
- Saga age metrics for stuck DEBITED detection
- Nightly reconciliation as last-line defense
- Game days testing AZ and cell failure quarterly
Key points
- Requirements first — CAP stance, latency SLA, read/write ratio, durability guarantees. ShardPay's ledger is CP; saying "we need ACID everywhere" without qualifying scope fails interviews.
- Failure modes per component — partition, crash, slow node, duplicate message, clock skew. Walk the data flow arrow by arrow, not as a generic list.
- Mitigation catalog — retries with backoff, idempotency keys, quorum reads, saga + compensation, circuit breaker + bulkhead, cell isolation. Match mitigation to specific failure, not spray-and-pray.
- Observability as proof — SLI/SLO, golden signals, traces, reconciliation. "We'd run a game day killing one AZ" demonstrates operational maturity.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Common traps and ShardPay answers
- Module map for quick recall
Test yourself
Answer these before moving on — recall is what makes it stick.