CAP Theorem — Theory and Practice

CAP in one partition

When two ledger replicas lose connectivity, they cannot both accept writes and still guarantee that every reader sees the same balance. If both accept writes (AP), balances diverge until reconciliation. If one side rejects writes (CP), some clients see errors but committed data stays consistent across surviving nodes.

ShardPay's ledger shard runs three replicas in one region. During an inter-AZ partition, the minority side stops accepting transfers and returns 503 ShardUnavailable while the majority continues committing. Merchants see a brief error banner; no account is debited twice across divergent copies.

Why partition tolerance is not optional

CAP is often misread as "pick two of three." In practice, networks partition and you still need the system to run. The real choice during a partition is between consistency and availability — not whether to tolerate partitions at all.

ShardPay's SRE runbook labels any shard where replica_reachable_count < quorum as partition mode. Dashboards switch from latency SLOs to "reject rate on minority" SLOs. On-call engineers do not try to "fix CAP" — they confirm the CP component is rejecting writes on the isolated minority and that AP components have reconciliation jobs running.

Walkthrough: AZ failure on a ledger shard

1. Shard ledger-42 has replicas in us-east-1a, 1b, and 1c.
2. A network misconfiguration isolates 1c from 1a and 1b.
3. The leader in 1a still has quorum (2 of 3) and continues commits.
4. The replica in 1c cannot reach quorum; it stops accepting writes and returns errors to any client routed there.
5. Load balancers health-check 1c out of the write pool within 10 seconds.
6. After the network heals, 1c catches up via Raft log replication — no manual merge.

// ShardPay ledger write gate — CP behavior during quorum loss
public WriteResult commitTransfer(Transfer transfer) {
    if (!replicationGroup.hasWriteQuorum()) {
        return WriteResult.rejected("SHARD_UNAVAILABLE",
            "Write quorum lost; refusing transfer to preserve consistency");
    }
    return leader.appendAndReplicate(transfer);
}

Designing CP vs AP per component

A single product is never purely CP or AP. ShardPay splits the architecture: financial state is CP; derived views and notifications are AP. The mistake is applying one global label instead of documenting each data path.

Example: Notification feed during partition

A merchant in eu-west-1 triggers a payout while the regional link to us-east-1 is down. The EU notification service writes the event to a local outbox and returns 200 OK immediately (AP). When the link recovers, a merger job replays EU events into the global stream, skipping duplicates by transferId. The merchant may have seen the notification before the US analytics pipeline counted the payout — both are correct under their consistency contracts.