Skip to main content
PreparationZone Logo
PrepZone

Distributed Locks and Fencing Tokens

Why naive Redis locks fail and how monotonic fencing tokens prevent stale leader writes.

Pass checkpoint below

Why this matters

  • Locks are easy to get wrong and hard to debug — paused GC, clock skew, and network delays create subtle double-execution.
  • Martin Kleppmann's fencing token article is interview gold — storage must validate tokens, not just the lock service.
  • ShardPay uses locks only for batch reconciliation jobs, not per-transfer — idempotency and single-leader writes cover the hot path.
  • Redlock and TTL-only patterns fail without fencing — know why before proposing locks in design reviews.
Old leaderToken 41 — rejected
StorageChecks token > last write
New leaderToken 42 — accepted
Monotonic token rejects writes from a leader that lost the lease.

Locks that actually work

ShardPay's end-of-day reconciliation job acquires a distributed lock before scanning unsettled transfers. The lock has a 30-second lease renewed every 10 seconds while the worker is healthy. The object store that receives reconciliation reports checks a monotonic fencing token on every write — a paused worker whose lease expired cannot overwrite a newer run's output.

Per-transfer locking was rejected: leader serialization plus idempotency keys achieve exclusivity without Redis in the payment path.

Lock primitives

  • Lease — Lock with TTL; holder must renew before expiry. ShardPay's reconciliation worker calls renewLease() on a heartbeat thread; missed renewal releases lock for standby.
  • Fencing token — Monotonic number issued with lock grant; storage rejects writes with token < highest seen. ShardPay's S3 writer stores fenceToken in object metadata; stale workers get 412 Precondition Failed.
  • Linearizable lock service — Lock grant must be consistent (etcd, ZooKeeper, Consul with CP semantics). ShardPay uses etcd transactional lock: create /locks/reconcile if not exists with lease.
  • Redlock critique — Multiple independent Redis masters without shared fencing — clock drift and delayed processes can double-acquire. ShardPay does not use Redlock for financial exclusivity; etcd + fencing for batch jobs only.
  • Prefer alternatives — Single Raft leader, idempotency keys, compare-and-set, database SELECT FOR UPDATE. ShardPay transfers use leader append + transferId dedup — no lock.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Safe lock pattern
  • When locks are and are not appropriate

Test yourself

Answer these before moving on — recall is what makes it stick.