Skip to main content
PreparationZone Logo
PrepZone

Circuit Breaker and Bulkhead

Fail fast and isolate thread pools so one slow dependency cannot exhaust ShardPay workers.

Pass checkpoint below

Why this matters

  • Without bulkheads, a hung fraud vendor blocked all ShardPay transfer threads — including transfers that could skip fraud in low-risk corridors.
  • Circuit breakers convert slow failures into fast failures, preserving capacity for healthy paths and graceful degradation.
  • Resilience4j and Istio implement these patterns in Java/Kubernetes stacks ShardPay runs in production.
  • Interviewers ask how you degrade when fraud is down without losing committed ledger state.
ClosedNormal traffic
OpenFail fast
Half-openProbe request
Open circuit fails fast; half-open probes recovery.

Circuit breaker states

A circuit breaker wraps outbound calls to a dependency. Closed: calls pass through; failures increment a sliding-window counter. Open: calls fail immediately without hitting the dependency — ShardPay returns pending_fraud from cache policy. Half-open: one probe request tests recovery; success closes the circuit, failure reopens.

ShardPay opens the fraud circuit when error rate exceeds 50% over a 30-second window with minimum 20 calls — avoiding flapping on single errors during low traffic.

Walkthrough: fraud vendor outage

At 14:00 fraud vendor begins returning 503 for all requests. By 14:00:20 ShardPay's circuit opens. Transfers from 14:00:20 to 14:05 complete ledger commit but skip sync fraud; async workers queue scoring for when half-open probes succeed. Ledger p99 stays under 200ms; manual review queue depth rises but the platform stays up. At 14:05 a probe succeeds; circuit closes and drains the backlog with rate limiting.

Key points

  • Closed state — normal operation; failures tracked in a rolling window. ShardPay logs circuit metrics to Grafana alongside fraud vendor status page correlation for post-incident review.
  • Open state — fail fast without calling the dependency; protects the vendor from retry amplification and protects ShardPay threads from blocking. Open fraud circuit routes transfers to pending_fraud with customer-visible "verification in progress" copy.
  • Half-open state — allows a single probe (or small probe percentage in Istio) to test recovery before restoring full traffic. ShardPay requires three consecutive probe successes before closing to avoid flapping on intermittent 200s.
  • Bulkhead — isolates resource pools per dependency so one slow callee cannot exhaust all threads. ShardPay's transfer executor uses 80 threads for ledger and a dedicated 20-thread pool for fraud RPCs — fraud saturation never blocks ledger-only maintenance jobs.
  • Graceful degradation — define product behavior when circuit is open: sync path vs async queue vs hard fail. ShardPay never rolls back committed ledger entries when fraud opens; compliance accepts async scoring within 15 minutes for low-value corridors.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Bulkheads and thread isolation
  • Service mesh vs library breakers

Test yourself

Answer these before moving on — recall is what makes it stick.