Latency, Throughput, and Little's Law

Latency vs throughput

Latency is the time for one operation to complete — the customer's wait from tap "Send" to confirmation. Throughput is operations completed per unit time — transfers per second across the fleet. They are related but not interchangeable: you can raise throughput by batching 100 transfers into one disk write, but each individual transfer waits longer in the batch queue.

ShardPay caps batch size at 50 transfers so p99 stays under 200ms. Without the cap, throughput graphs looked healthy while customers abandoned slow confirmations. Optimizing one metric in isolation often harms the other.

Little's Law in practice

Little's Law is deceptively simple but powerful. You do not need to know queue discipline or service time distribution — if you measure any two of L, λ, and W, you get the third. During a Black Friday spike, ShardPay's transfer queue depth jumped from 50 to 800. Arrival rate was 3,500/sec. Little's Law predicted average wait: 800 / 3500 ≈ 229ms — matching the observed p50 latency spike without any code profiling.

The law also explains why adding capacity does not always help. If the bottleneck is a single leader shard, adding API pods increases λ to the shard without reducing W. The queue grows on the shard, not the API tier.

Walkthrough: finding the hidden queue

ShardPay's dashboard showed API CPU at 40% and ledger CPU at 55% — both looked healthy. But transfer p99 hit 450ms. Little's Law on the ledger write queue: L = 900, λ = 2,000/sec → W = 450ms. The bottleneck was disk fsync on the leader, not CPU. Adding API instances made it worse by increasing λ. The fix was faster disks and write batching, not more pods.

// ShardPay metrics — expose queue depth for Little's Law monitoring
@Component
public class TransferQueueMetrics {
    private final AtomicInteger queueDepth = new AtomicInteger();

    public void enqueue(Transfer transfer) {
        queueDepth.incrementAndGet();
        ledgerPool.submit(() -> {
            process(transfer);
            queueDepth.decrementAndGet();
        });
    }

    @Scheduled(fixedRate = 10_000)
    public void reportQueueStats() {
        int depth = queueDepth.get();
        double arrivalRate = rateCounter.transfersPerSecond();  // λ
        if (arrivalRate > 0) {
            double predictedWaitMs = (depth / arrivalRate) * 1000;  // W = L/λ
            metrics.gauge("transfer.queue.predicted_wait_ms", predictedWaitMs);
            metrics.gauge("transfer.queue.depth", depth);
        }
    }
}

Tail latency and the fan-out problem

When a request fans out to N shards or services, the overall latency is dominated by the slowest responder, not the average. With 10 parallel calls each at p99 = 50ms, the probability that at least one exceeds 50ms approaches certainty as N grows. ShardPay limits fan-out to three shards per transfer and uses deadline propagation so a slow shard does not consume the entire budget.

Coordinated omission is another tail trap: if the client times out at 200ms, it stops recording latency for requests that would have taken 500ms. Your p99 looks artificially low. ShardPay's load generator records "timeout" as 200ms+ in histograms to avoid lying to itself.

Optimizing the bottleneck

Improving non-bottleneck components wastes effort. ShardPay's transfer path is: gateway (15ms) → auth (20ms) → ledger (80ms) → fraud (40ms) → response. Optimizing auth from 20ms to 10ms saves 10ms; optimizing ledger from 80ms to 50ms saves 30ms. The ledger is the bottleneck — every optimization dollar goes there first.