Why this matters
- Architects are judged on blast radius containment, not just uptime percentages.
- Correlated failures — same software version, shared control plane, single power feed — defeat naive N+1 redundancy.
- ShardPay's 2024 cell-1 incident affected 12,000 merchants, not 1.2M, because cell boundaries contained the blast.
- Staff+ interviews ask you to identify hidden shared failure domains in a diagram, not just label AZs.
Mapping failure domains
A failure domain is any scope where one trigger (power loss, network cut, bad config, CVE exploit) can knock out multiple components at once. Blast radius is how much of the business stops working when that domain fails.
ShardPay maps domains hierarchically: rack → AZ → region → cell → global control plane. Each layer should contain failures from the layer below without cascading upward unchecked.
Key points
- Failure domain — the scope of a single root-cause event. ShardPay treats one AZ as a failure domain: losing
us-east-1ashould not take downus-east-1borus-east-1c. - Blast radius — the user-visible impact when a domain fails. Cell 1 outage affects merchants 1–12,000; shared identity service outage affects all 1.2M merchants — a larger blast radius by design (with extra redundancy).
- Physical isolation — AZs have independent power and networking within a region. ShardPay runs each shard's replicas across 3 AZs with quorum writes requiring 2/3 AZs.
- Correlated failure — same bug in every instance because every instance runs the same deploy. ShardPay uses canary deploys per cell so a bad binary reaches cell 1 before cell 4.
ShardPay isolation strategy
ShardPay combines AZ redundancy within cells and cell independence across the fleet. The remaining intentional shared domains are identity, DNS, and the deployment pipeline — each with extra hardening.
Per-shard AZ placement
Each ledger shard has a primary in one AZ and replicas in two others. Write quorum (2 of 3) survives single-AZ loss. ShardPay tested this with AZ failure drills quarterly — last drill recovered write path in 47 seconds.
Cell independence
Four cells each serve ~300k merchants with full stack: load balancers, API, orchestrator, shards, Kafka cluster. Cell 2's Kafka cluster is separate from cell 1's — no cross-cell broker dependency during normal operation.
Shared control plane risks
Identity (OAuth tokens), external DNS, and the CI/CD pipeline are shared across cells. These are ShardPay's largest blast radius by design. Mitigations: multi-AZ identity replicas, DNS with anycast failover, deploy gates requiring per-cell canary success.
// ShardPay cell-aware routing — blast radius containment
public Cell routeMerchant(String merchantId) {
int cellId = consistentHash(merchantId, cellRegistry.activeCells());
Cell cell = cellRegistry.get(cellId);
if (cell.health() == CellHealth.DEGRADED) {
// Fail over only within pre-defined pairs — not global drain
return cellRegistry.failoverPartner(cellId);
}
return cell;
}
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 1 section
- Reducing blast radius over time
Test yourself
Answer these before moving on — recall is what makes it stick.