RTT and the Bandwidth-Delay Product

Physics of the network

Light travels ~200,000 km/s in fiber. A round trip between US coasts (~5,000 km) takes ~50ms minimum — before switches, queuing, TLS handshakes, or application processing. ShardPay cannot strong-consensus every transfer across regions without paying that tax on every write. This is not a software problem; it is physics.

Bandwidth is how wide the pipe is (bits per second). Latency (RTT) is how long a single bit takes to make a round trip. They are independent dimensions. A 10 Gbps link with 70ms RTT does not deliver 10 Gbps of throughput for small messages — each message waits 70ms for acknowledgment before the next can be sent (without pipelining).

Why cross-region consistency is expensive

ShardPay's EU expansion required a regional ledger cell. The alternative — strong-consensus every write across US and EU — would add 130ms RTT (US-East to EU-West) to every transfer. At 5,000 transfers/sec, that is 5,000 synchronous cross-Atlantic round trips per second, consuming bandwidth and making p99 latency unacceptable.

The solution: regional cells with async cross-region replication for disaster recovery, not per-write consensus. RPO for cross-region is 5 seconds (async replication lag), not zero. This is a deliberate trade-off: durability within a region (zero RPO) vs cross-region (seconds of RPO) to avoid paying RTT on every write.

Walkthrough: replication throughput math

ShardPay replicates ledger entries from US-East leader to US-West follower. Link: 10 Gbps, RTT: 70ms. Entry size: 2 KB.

Without pipelining: 1 entry per RTT = 2 KB / 70ms ≈ 28 KB/s ≈ 0.22 Mbps — using 0.002% of the 10 Gbps pipe.

With pipelining (64 entries in flight): 64 × 2 KB / 70ms ≈ 1.8 MB/s ≈ 14 Mbps — still only 0.14% utilization.

With batching (50 entries per message, 64 batches in flight): 64 × 50 × 2 KB / 70ms ≈ 91 MB/s ≈ 730 Mbps — approaching useful utilization.

// ShardPay replication client — pipeline + batch to overcome RTT
public class ReplicationClient {
    private static final int PIPELINE_DEPTH = 64;
    private static final int BATCH_SIZE = 50;

    private final BlockingQueue<LedgerEntry> pending = new LinkedBlockingQueue<>();
    private final Channel channel;  // gRPC to follower region

    public void replicate(LedgerEntry entry) {
        pending.offer(entry);
    }

    @Scheduled(fixedDelay = 5)  // flush every 5ms or when batch full
    public void flushBatch() {
        List<LedgerEntry> batch = new ArrayList<>(BATCH_SIZE);
        pending.drainTo(batch, BATCH_SIZE);
        if (!batch.isEmpty()) {
            channel.replicateBatch(batch);  // one RTT for up to 50 entries
        }
    }
}

Latency budgets across regions

ShardPay's 200ms transfer SLO cannot survive a 130ms network round trip for every hop. Regional cells keep the critical path within one region (~5ms RTT). Cross-region traffic is async: backup replication, analytics export, and disaster-recovery failover — never on the synchronous transfer path.

When designing multi-region systems, map every synchronous call on the critical path and sum the RTTs. If the sum exceeds the SLA, move calls off the critical path (async replication, regional cells, edge caching) or accept higher latency.