Skip to main content
PreparationZone Logo
PrepZone

The Distributed Systems Interview Playbook

Framework for trade-off questions, failure scenarios, and architect-level system reasoning.

Pass checkpoint below

Why this matters

  • This article consolidates the Distributed Systems track into an interview-ready framework.
  • Interviewers at senior/staff level rarely ask trivia — they present ambiguous problems and evaluate structured thinking.
  • ShardPay examples from modules 1–12 provide concrete anchors for every abstract trade-off.
  • Revisit this article after completing all modules for spaced recall before interview loops.
ClientMobile / web
API gateway
Ledger AShard 1
Ledger BShard 2
Ledger CShard 3
Independent nodes communicate over a network. Any node can fail without warning.

The four-step framework

Every distributed systems design answer at ShardPay interview bar follows four steps. Skip any step and the answer feels junior.

Step 1: Clarify requirements

Before drawing boxes, state assumptions:

  • Scale: 1M merchants, 3M transfers/day, peak 500/sec
  • Consistency: ledger CP (no double-spend); activity feed AP
  • Latency: p99 transfer < 200ms; cross-shard acceptable at 400ms
  • Durability: no acknowledged transfer lost; reconciliation within 24h
  • Availability: 99.95% transfer SLO; brief read staleness OK for dashboards

Ask the interviewer which dimension matters most if time is short. "If I can only optimize one property, should it be correctness or latency?"

Step 2: Draw components and data flow

Sketch: API gateway → fraud → orchestrator → shard A debit → shard B credit → outbox → settlement consumer. Label sync vs async, CP vs AP per component.

Name ShardPay's actual choices: orchestrated saga (not 2PC), consistent hash sharding, cell-based routing, Kafka at-least-once with idempotent consumers.

Step 3: Name failure modes

For each arrow in the diagram, ask "what if this fails?"

FailureImpactShardPay mitigation
Network partition between shardsSaga stuck in DEBITEDStep timeout + compensation
Coordinator crashSaga pause, not duplicateDurable saga log + leader election
Duplicate Kafka eventDouble credit riskIdempotency key on consumer
Hot shardp99 latency spikeConsistent hash + sub-shards
AZ outage1/3 replica lossQuorum 2/3 writes continue
Bad deployRegression in transfer pathCell-scoped canary + rollback
  • Network partition between shards

    ImpactSaga stuck in DEBITED
    ShardPay mitigationStep timeout + compensation
  • Coordinator crash

    ImpactSaga pause, not duplicate
    ShardPay mitigationDurable saga log + leader election
  • Duplicate Kafka event

    ImpactDouble credit risk
    ShardPay mitigationIdempotency key on consumer
  • Hot shard

    Impactp99 latency spike
    ShardPay mitigationConsistent hash + sub-shards
  • AZ outage

    Impact1/3 replica loss
    ShardPay mitigationQuorum 2/3 writes continue
  • Bad deploy

    ImpactRegression in transfer path
    ShardPay mitigationCell-scoped canary + rollback

Common failure modes and mitigations

Step 4: Propose mitigations and observability

Mitigations without observability are unverifiable. Every ShardPay answer ends with: "I'd know this works because..."

  • Transfer SLI at gateway with 99.95% SLO and error budget alerts
  • Distributed traces with transferId baggage across all saga steps
  • Saga age metrics for stuck DEBITED detection
  • Nightly reconciliation as last-line defense
  • Game days testing AZ and cell failure quarterly

Key points

  • Requirements first — CAP stance, latency SLA, read/write ratio, durability guarantees. ShardPay's ledger is CP; saying "we need ACID everywhere" without qualifying scope fails interviews.
  • Failure modes per component — partition, crash, slow node, duplicate message, clock skew. Walk the data flow arrow by arrow, not as a generic list.
  • Mitigation catalog — retries with backoff, idempotency keys, quorum reads, saga + compensation, circuit breaker + bulkhead, cell isolation. Match mitigation to specific failure, not spray-and-pray.
  • Observability as proof — SLI/SLO, golden signals, traces, reconciliation. "We'd run a game day killing one AZ" demonstrates operational maturity.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Common traps and ShardPay answers
  • Module map for quick recall

Test yourself

Answer these before moving on — recall is what makes it stick.