The Distributed Systems Interview Playbook

The four-step framework

Every distributed systems design answer at ShardPay interview bar follows four steps. Skip any step and the answer feels junior.

Step 1: Clarify requirements

Before drawing boxes, state assumptions:

• Scale: 1M merchants, 3M transfers/day, peak 500/sec
• Consistency: ledger CP (no double-spend); activity feed AP
• Latency: p99 transfer < 200ms; cross-shard acceptable at 400ms
• Durability: no acknowledged transfer lost; reconciliation within 24h
• Availability: 99.95% transfer SLO; brief read staleness OK for dashboards

Ask the interviewer which dimension matters most if time is short. "If I can only optimize one property, should it be correctness or latency?"

Step 2: Draw components and data flow

Sketch: API gateway → fraud → orchestrator → shard A debit → shard B credit → outbox → settlement consumer. Label sync vs async, CP vs AP per component.

Name ShardPay's actual choices: orchestrated saga (not 2PC), consistent hash sharding, cell-based routing, Kafka at-least-once with idempotent consumers.

Step 3: Name failure modes

For each arrow in the diagram, ask "what if this fails?"

Step 4: Propose mitigations and observability

Mitigations without observability are unverifiable. Every ShardPay answer ends with: "I'd know this works because..."

• Transfer SLI at gateway with 99.95% SLO and error budget alerts
• Distributed traces with transferId baggage across all saga steps
• Saga age metrics for stuck DEBITED detection
• Nightly reconciliation as last-line defense
• Game days testing AZ and cell failure quarterly

Common traps and ShardPay answers

Interview traps recur across companies. Prepare one-paragraph responses anchored in this track's examples.

"We'll use 2PC for cross-shard transfers"

Trap: atomicity without discussing blocking and partition intolerance.

ShardPay answer: "We retired XA in 2023 after an 11-minute lock outage. Cross-shard transfers use orchestrated sagas with compensating debits, participant idempotency keys, and nightly reconciliation. We accept brief intermediate inconsistency bounded by 30-second step timeouts."

"Exactly-once delivery"

Trap: claiming end-to-end exactly-once without idempotency at every side effect.

ShardPay answer: "Kafka gives at-least-once; we achieve effective exactly-once with idempotent producers, consumer dedup by eventId, and Idempotency-Key on transfer API. Exactly-once is a pipeline property, not a broker toggle."

"Microservices are always better"

Trap: ignoring operational cost and distributed debugging complexity.

ShardPay answer: "We split along team boundaries (Conway) into ledger, fraud, merchant, settlement stream teams. Each service has on-call, SLO, and deploy pipeline. The monolith was correct at 10k merchants; cells + services at 1M. Operational cost is 4 on-call rotations and cross-service tracing — worth it at our scale."

// Interview snippet — tie idempotency + saga + observability together
@PostMapping("/transfers")
public ResponseEntity<TransferResponse> createTransfer(
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestBody TransferRequest req) {

    Optional<TransferResponse> cached = idempotencyStore.get(idempotencyKey);
    if (cached.isPresent()) return ResponseEntity.ok(cached.get());

    Span span = tracer.spanBuilder("transfer.create")
        .setAttribute("merchant.id", req.merchantId())
        .startSpan();
    try (Scope s = span.makeCurrent()) {
        SagaInstance saga = orchestrator.start(req, idempotencyKey);
        TransferResponse resp = TransferResponse.from(saga);
        idempotencyStore.put(idempotencyKey, resp);
        sliRecorder.recordTransfer(resp, span);
        return ResponseEntity.accepted().body(resp);
    } finally {
        span.end();
    }
}

Module map for quick recall

Link interview topics to track modules for deeper review: