Skip to main content
PreparationZone Logo
PrepZone

Hot Spots, Skew, and Rebalancing

Celebrity accounts and power-law traffic — detect skew and rebalance without downtime.

Pass checkpoint below

Why this matters

  • Horizontal scale fails if one shard absorbs 40% of traffic — adding nodes does not help skewed keys.
  • Celebrity accounts, platform settlement wallets, and payroll batch windows are classic ShardPay hot spots.
  • Rebalancing is an operational discipline with dual-write cutovers — not a one-time cluster setup task.
  • Interviewers ask how you detect skew and migrate without downtime or data loss.
Hash ring
Node Avn1, vn2
Node Bvn3, vn4
Node Cvn5, vn6

Keys hash onto the ring; walk clockwise to find owner

Virtual nodes spread keys evenly. Adding a node moves only adjacent key ranges.

Detect and fix skew

ShardPay monitors per-shard QPS, p99 latency, and disk growth every minute. A merchant payroll day spiked shard ledger-42 to 18k TPS while fleet median was 3k — PagerDuty fired on shard_qps_skew_ratio > 3. Root cause: enterprise tenant's 50,000 employee accounts hashed across shards, but all debits targeted one platform settlement account on ledger-42.

Fix: split the settlement account into 16 salted sub-accounts (settle-{00..0f}) with scatter-gather aggregation on admin reads.

Skew vocabulary

  • Skew — Uneven distribution of keys, traffic, or storage across partitions. ShardPay dashboards rank shards by qps / fleet_avg_qps — anything above 2.0 enters investigation.
  • Hot key — Single key receiving disproportionate access. ShardPay's platform:settlement:usd row received 40% of write IOPS on one shard despite even account hashing.
  • Hot partition / shard — Entire shard overloaded because many hot keys or one dominant tenant landed there. Directory override moved a SaaS tenant to dedicated hardware after composite key isolation was insufficient.
  • Split — Divide a partition or key into children (sub-keys, sub-shards). ShardPay splits hot logical shards by doubling vnode density on the ring segment and migrating half the key range to new hardware.
  • Migrate — Copy key range from source to destination with dual-write and verification. ShardPay's migration controller runs copy → dual-write → verify checksum → cutover → retire source.
  • Salting — Append random prefix to spread logical key across multiple physical keys (acc-123#03). Read path merges — ShardPay uses salting only when write QPS exceeds single-row limits; reads pay scatter-gather cost.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Detection signals
  • Mitigation strategies

Test yourself

Answer these before moving on — recall is what makes it stick.