Distributed ID and Sequencer Design

Snowflake-style IDs at ShardPay

ShardPay encodes every transfer ID as 64 bits: 41-bit millisecond timestamp, 10-bit shard ID, 12-bit per-shard sequence. IDs are roughly time-ordered, unique cluster-wide, and decodable for debugging ("this transfer landed on shard 42 at 14:03 UTC").

Each ledger shard allocates sequence numbers locally until the millisecond bucket rolls over or the 12-bit sequence exhausts (4096/ms). On exhaustion the shard spins until the next millisecond — rare at ShardPay's per-shard throughput but load-tested to 4k transfers/ms per shard.

Walkthrough: support lookup by time range

A merchant disputes all transfers between 2pm and 3pm. Because IDs embed timestamp, ShardPay's support API computes min/max ID bounds for that window and queries only relevant shards. Without time-ordered IDs the team would full-scan UUID primary keys — unusable at billions of rows.

Raft-backed centralized sequencers

When strict monotonic sequence numbers must be global — regulatory export batch IDs, exactly-once settlement file sequence — ShardPay runs a small Raft cluster (three nodes) that increments a counter only after majority commit. Clients fetch batches of 1000 IDs per RPC to amortize consensus latency.

The sequencer is a SPOF in theory but not in practice: Raft tolerates one node loss; clients retry on leader election blips. Throughput is lower than Snowflake (roughly 50k IDs/s cluster-wide) so ShardPay reserves it for low-volume, high-assurance sequences.

Walkthrough: leader failover during peak

During Black Friday, the Raft leader for the settlement sequencer dies. Followers elect a new leader in ~300ms. Three in-flight settlement workers receive NOT_LEADER and retry with backoff; none duplicate batch IDs because uncommitted leader proposals are discarded. Settlement files remain strictly ordered.

// ShardPay Snowflake generator on a ledger shard (Java 17)
public final class SnowflakeTransferIdGenerator {
    private static final int SHARD_BITS = 10;
    private static final int SEQ_BITS = 12;
    private static final long MAX_SEQUENCE = (1L << SEQ_BITS) - 1;

    private final int shardId;
    private long lastMillis = -1;
    private long sequence = 0;

    public SnowflakeTransferIdGenerator(int shardId) {
        if (shardId < 0 || shardId >= (1 << SHARD_BITS)) {
            throw new IllegalArgumentException("shardId out of range");
        }
        this.shardId = shardId;
    }

    public synchronized long nextId() {
        long now = System.currentTimeMillis();
        if (now == lastMillis) {
            sequence = (sequence + 1) & MAX_SEQUENCE;
            if (sequence == 0) {
                now = waitNextMillis(now);
            }
        } else {
            sequence = 0;
        }
        lastMillis = now;
        return (now << (SHARD_BITS + SEQ_BITS))
            | ((long) shardId << SEQ_BITS)
            | sequence;
    }

    private long waitNextMillis(long current) {
        while (System.currentTimeMillis() <= current) { /* spin */ }
        return System.currentTimeMillis();
    }
}

Choosing an ID strategy

Match the scheme to query patterns and failure tolerance. ShardPay's decision matrix: hot-path transfer IDs → Snowflake on shard; global batch sequence → Raft sequencer; one-off correlation → UUID v4 in HTTP headers. Mixing schemes in one table is fine; mixing semantics (using Snowflake order to infer causality) is not.