Why this matters
- Clock skew caused duplicate transfer IDs at ShardPay before logical clocks replaced timestamp-only ordering.
- NTP synchronizes wall clocks but does not guarantee monotonicity or cross-node ordering at payment latency scales.
- Sorting audit logs by
created_atmisorders causally related ledger entries during daylight-saving jumps and VM clock steps. - Interviewers probe whether you know when physical time is safe (TTL expiry) versus when you need logical or centralized ordering.
Why wall clocks lie
Two ShardPay ledger nodes can disagree on "now" by hundreds of milliseconds even with NTP. When node A records transfer T1 at 09:00:00.800 and node B records T2 at 09:00:00.300, sorting by wall clock shows T2 before T1 even if T1 caused T2. That breaks dispute investigations and regulatory audit trails.
Physical clocks also jump backward after leap seconds, VM migrations, and manual corrections. A monotonic clock (System.nanoTime()) avoids backward jumps on one JVM but still cannot order events across machines.
Walkthrough: the audit replay bug
ShardPay's compliance team replayed March settlement logs sorted by event_timestamp. Two refunds for the same merchant appeared in reverse causal order because the processing node had drifted 600ms ahead. The replay flagged a false fraud pattern. The fix was to attach Lamport timestamps at write time and sort by (lamport, node_id) instead of wall clock alone.
Key points
- Clock skew — the difference between two nodes' notion of current time. At ShardPay scale even 200ms skew can invert the order of back-to-back transfers on different shards; never use wall clock as the sole ordering key for correlated events.
- Clock drift — the rate at which a clock gains or loses time relative to UTC. Drift accumulates between NTP corrections; ShardPay monitors drift per host and alerts when offset exceeds 100ms so ops can replace bad hardware before ordering bugs surface.
- NTP — Network Time Protocol pulls nodes toward a stratum source but only guarantees bounded offset, not strict monotonicity. ShardPay runs
chronyon every ledger pod; NTP is for human-readable timestamps and TTL, not for cross-shard causal ordering. - Monotonic clock — per-process counter that never decreases, useful for measuring elapsed time on one JVM. ShardPay uses
Instantfor API latency budgets but pairs it with logical clocks when events must be compared across nodes. - Hybrid Logical Clock (HLC) — combines physical time with a logical counter so timestamps stay roughly aligned with real time while preserving happened-before. ShardPay's cross-region audit aggregator uses HLC when merging logs from three regions without a single sequencer bottleneck.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Ordering strategies at ShardPay
- When physical time is enough
Test yourself
Answer these before moving on — recall is what makes it stick.