Why this matters
- ShardPay's untested failover path for shard metadata lost routing table updates during the first real AZ failure.
- Jepsen history analysis exposed "linearizable" claims that lost committed transfers under network partition in other systems — ShardPay runs similar tests on ledger replicas before major upgrades.
- Game days reveal retry policies, circuit breakers, and runbooks that unit tests never exercise.
- Interviewers ask how you validate consistency and resilience claims without waiting for production incidents.
Chaos experiments with hypothesis
A chaos experiment starts with a hypothesis: "If we kill one ledger pod per shard during peak load, p99 transfer latency stays under 300ms and no committed transfer is lost." ShardPay defines blast radius (staging only, one region), abort conditions (error rate > 5%, manual kill switch), and observability dashboards before injecting failure.
Tools: Litmus or Chaos Mesh on Kubernetes for pod kills; Toxiproxy for latency and partition simulation; custom scripts for clock skew injection on staging nodes.
Walkthrough: monthly game day
ShardPay's SRE team runs a four-hour game day in staging mirroring production traffic shape. At T+60m they terminate 30% of ledger pods randomly. Hypothesis holds except fraud circuit opens from retry noise — runbook updated to widen bulkhead before next drill. At T+120m they partition a minority of Raft voters from the sequencer cluster; no duplicate settlement batch IDs; one 400ms leader election blip logged.
Key points
- Chaos experiment — structured test with hypothesis, metrics, blast radius, and abort criteria. ShardPay files a one-page experiment doc in Confluence before every drill; post-mortem compares expected vs actual behavior.
- Game day — scheduled multi-team drill simulating realistic failures under load. ShardPay includes payments ops, support, and comms to practice customer messaging when transfers enter
pending_fraudat scale. - Jepsen — framework that injects network partitions and clock skew while a workload generator runs; analyzes history for linearizability and other consistency violations. ShardPay adapted Jepsen-style workloads against staging ledger replicas before migrating from sync to semi-sync replication.
- Blast radius — limit scope: one namespace, one region, max percentage of pods. Production chaos at ShardPay stays at ≤1% canary traffic with instant rollback — never "random prod kills" without guardrails.
- Consistency checking — compare linearization graphs from concurrent read/write histories. ShardPay's checker verifies no lost committed transfer after partition heals — if Jepsen finds an anomaly, release blocks.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Jepsen-style verification for the ledger
- From staging chaos to production confidence
Test yourself
Answer these before moving on — recall is what makes it stick.