Skip to main content
PreparationZone Logo
PrepZone

Physical vs Logical Clocks

Wall clocks lie across datacenters — why ShardPay never orders events by `System.currentTimeMillis()`.

Pass checkpoint below

Why this matters

  • Clock skew caused duplicate transfer IDs at ShardPay before logical clocks replaced timestamp-only ordering.
  • NTP synchronizes wall clocks but does not guarantee monotonicity or cross-node ordering at payment latency scales.
  • Sorting audit logs by created_at misorders causally related ledger entries during daylight-saving jumps and VM clock steps.
  • Interviewers probe whether you know when physical time is safe (TTL expiry) versus when you need logical or centralized ordering.
Event AL=1
Event BL=2
Event CL=2
Lamport timestamps establish happened-before without wall-clock sync.

Why wall clocks lie

Two ShardPay ledger nodes can disagree on "now" by hundreds of milliseconds even with NTP. When node A records transfer T1 at 09:00:00.800 and node B records T2 at 09:00:00.300, sorting by wall clock shows T2 before T1 even if T1 caused T2. That breaks dispute investigations and regulatory audit trails.

Physical clocks also jump backward after leap seconds, VM migrations, and manual corrections. A monotonic clock (System.nanoTime()) avoids backward jumps on one JVM but still cannot order events across machines.

Walkthrough: the audit replay bug

ShardPay's compliance team replayed March settlement logs sorted by event_timestamp. Two refunds for the same merchant appeared in reverse causal order because the processing node had drifted 600ms ahead. The replay flagged a false fraud pattern. The fix was to attach Lamport timestamps at write time and sort by (lamport, node_id) instead of wall clock alone.

Key points

  • Clock skew — the difference between two nodes' notion of current time. At ShardPay scale even 200ms skew can invert the order of back-to-back transfers on different shards; never use wall clock as the sole ordering key for correlated events.
  • Clock drift — the rate at which a clock gains or loses time relative to UTC. Drift accumulates between NTP corrections; ShardPay monitors drift per host and alerts when offset exceeds 100ms so ops can replace bad hardware before ordering bugs surface.
  • NTP — Network Time Protocol pulls nodes toward a stratum source but only guarantees bounded offset, not strict monotonicity. ShardPay runs chrony on every ledger pod; NTP is for human-readable timestamps and TTL, not for cross-shard causal ordering.
  • Monotonic clock — per-process counter that never decreases, useful for measuring elapsed time on one JVM. ShardPay uses Instant for API latency budgets but pairs it with logical clocks when events must be compared across nodes.
  • Hybrid Logical Clock (HLC) — combines physical time with a logical counter so timestamps stay roughly aligned with real time while preserving happened-before. ShardPay's cross-region audit aggregator uses HLC when merging logs from three regions without a single sequencer bottleneck.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Ordering strategies at ShardPay
  • When physical time is enough

Test yourself

Answer these before moving on — recall is what makes it stick.