Skip to main content
PreparationZone Logo
PrepZone

Retries, Backoff, and Jitter

Retry storms take down healthy services — exponential backoff with full jitter.

Pass checkpoint below

Why this matters

  • A retry storm during a 30-second fraud outage took down ShardPay's healthy ledger shards by multiplying traffic 10×.
  • Payment APIs without idempotency keys double-charge when clients retry POST on timeout.
  • Not every error is retriable — retrying 400 InsufficientFunds hammering the ledger wastes capacity and logs noise.
  • Interviewers ask for a full retry policy: which status codes, backoff shape, max attempts, and idempotency.
Retry delays
Attempt 10ms
Attempt 2~100ms + jitter
Attempt 3~200ms + jitter
Attempt 4~400ms + jitter
Spread retries to prevent thundering herd on recovery.

Retriable vs non-retriable errors

ShardPay classifies errors before retrying. Retriable: HTTP 408, 429, 502, 503, 504; gRPC UNAVAILABLE, DEADLINE_EXCEEDED; JDBC transient connection errors. Non-retriable: 400, 401, 403, 404, 409, 422; gRPC INVALID_ARGUMENT, FAILED_PRECONDITION; business errors like insufficient funds.

Ambiguous cases matter: a client timeout after the server committed must retry with the same idempotency key, not as a new transfer. ShardPay documents POST /transfers as safe to retry only when Idempotency-Key header is present.

Walkthrough: client timeout after commit

Mobile app POSTs a transfer; gateway times out at 200ms while ledger already committed. User taps "retry." Without idempotency the second POST creates a duplicate transfer. With Idempotency-Key: uuid-v4 the second request hits ShardPay's dedup store and returns the original transferId and 201 body — no double debit.

Key points

  • Exponential backoff — delay doubles each attempt (100ms, 200ms, 400ms) to give failing dependencies time to recover. ShardPay's Java RetryTemplate starts at 100ms base for ledger RPCs with max 3 attempts inside the parent deadline.
  • Full jitter — each delay is random(0, calculated_backoff) so synchronized clients desynchronize. After ShardPay added full jitter to fraud client retries, recovery time from vendor blips dropped because request peaks flattened.
  • Retry budget — cap total retry wall time (e.g. 150ms of retries inside 200ms SLA) and max attempt count. ShardPay aborts retries when Deadline.afterNow() would exceed gateway budget and returns pending state instead.
  • Idempotent retry — executing the same logical operation twice produces one durable effect. ShardPay requires idempotency keys on all mutating payment APIs; retries without keys get 428 Precondition Required.
  • Retry amplification — each client retry × each middleware retry × each downstream retry multiplies load. ShardPay disables automatic HTTP client retries on the gateway→ledger path; only the orchestrator retries with explicit policy.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Backoff configuration in practice
  • When not to retry

Test yourself

Answer these before moving on — recall is what makes it stick.