Why this matters
- Locks are easy to get wrong and hard to debug — paused GC, clock skew, and network delays create subtle double-execution.
- Martin Kleppmann's fencing token article is interview gold — storage must validate tokens, not just the lock service.
- ShardPay uses locks only for batch reconciliation jobs, not per-transfer — idempotency and single-leader writes cover the hot path.
- Redlock and TTL-only patterns fail without fencing — know why before proposing locks in design reviews.
Locks that actually work
ShardPay's end-of-day reconciliation job acquires a distributed lock before scanning unsettled transfers. The lock has a 30-second lease renewed every 10 seconds while the worker is healthy. The object store that receives reconciliation reports checks a monotonic fencing token on every write — a paused worker whose lease expired cannot overwrite a newer run's output.
Per-transfer locking was rejected: leader serialization plus idempotency keys achieve exclusivity without Redis in the payment path.
Lock primitives
- Lease — Lock with TTL; holder must renew before expiry. ShardPay's reconciliation worker calls
renewLease()on a heartbeat thread; missed renewal releases lock for standby. - Fencing token — Monotonic number issued with lock grant; storage rejects writes with token < highest seen. ShardPay's S3 writer stores
fenceTokenin object metadata; stale workers get412 Precondition Failed. - Linearizable lock service — Lock grant must be consistent (etcd, ZooKeeper, Consul with CP semantics). ShardPay uses etcd transactional lock:
create /locks/reconcile if not exists with lease. - Redlock critique — Multiple independent Redis masters without shared fencing — clock drift and delayed processes can double-acquire. ShardPay does not use Redlock for financial exclusivity; etcd + fencing for batch jobs only.
- Prefer alternatives — Single Raft leader, idempotency keys, compare-and-set, database
SELECT FOR UPDATE. ShardPay transfers use leader append +transferIddedup — no lock.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Safe lock pattern
- When locks are and are not appropriate
Test yourself
Answer these before moving on — recall is what makes it stick.