Skip to main content
PreparationZone Logo
PrepZone

Failure Domains and Blast Radius

Isolate AZs, regions, and cells so one incident cannot drain every ShardPay shard.

Pass checkpoint below

Why this matters

  • Architects are judged on blast radius containment, not just uptime percentages.
  • Correlated failures — same software version, shared control plane, single power feed — defeat naive N+1 redundancy.
  • ShardPay's 2024 cell-1 incident affected 12,000 merchants, not 1.2M, because cell boundaries contained the blast.
  • Staff+ interviews ask you to identify hidden shared failure domains in a diagram, not just label AZs.
Region us-east
Cell AAZ-1 + AZ-2
Cell BAZ-1 + AZ-2
Isolate blast radius by AZ, region, and cell.

Mapping failure domains

A failure domain is any scope where one trigger (power loss, network cut, bad config, CVE exploit) can knock out multiple components at once. Blast radius is how much of the business stops working when that domain fails.

ShardPay maps domains hierarchically: rack → AZ → region → cell → global control plane. Each layer should contain failures from the layer below without cascading upward unchecked.

Key points

  • Failure domain — the scope of a single root-cause event. ShardPay treats one AZ as a failure domain: losing us-east-1a should not take down us-east-1b or us-east-1c.
  • Blast radius — the user-visible impact when a domain fails. Cell 1 outage affects merchants 1–12,000; shared identity service outage affects all 1.2M merchants — a larger blast radius by design (with extra redundancy).
  • Physical isolation — AZs have independent power and networking within a region. ShardPay runs each shard's replicas across 3 AZs with quorum writes requiring 2/3 AZs.
  • Correlated failure — same bug in every instance because every instance runs the same deploy. ShardPay uses canary deploys per cell so a bad binary reaches cell 1 before cell 4.

ShardPay isolation strategy

ShardPay combines AZ redundancy within cells and cell independence across the fleet. The remaining intentional shared domains are identity, DNS, and the deployment pipeline — each with extra hardening.

Per-shard AZ placement

Each ledger shard has a primary in one AZ and replicas in two others. Write quorum (2 of 3) survives single-AZ loss. ShardPay tested this with AZ failure drills quarterly — last drill recovered write path in 47 seconds.

Cell independence

Four cells each serve ~300k merchants with full stack: load balancers, API, orchestrator, shards, Kafka cluster. Cell 2's Kafka cluster is separate from cell 1's — no cross-cell broker dependency during normal operation.

Shared control plane risks

Identity (OAuth tokens), external DNS, and the CI/CD pipeline are shared across cells. These are ShardPay's largest blast radius by design. Mitigations: multi-AZ identity replicas, DNS with anycast failover, deploy gates requiring per-cell canary success.

Java
// ShardPay cell-aware routing — blast radius containment
public Cell routeMerchant(String merchantId) {
    int cellId = consistentHash(merchantId, cellRegistry.activeCells());
    Cell cell = cellRegistry.get(cellId);
    if (cell.health() == CellHealth.DEGRADED) {
        // Fail over only within pre-defined pairs — not global drain
        return cellRegistry.failoverPartner(cellId);
    }
    return cell;
}

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 1 section

  • Reducing blast radius over time

Test yourself

Answer these before moving on — recall is what makes it stick.