Reliability, Availability, and Durability

Definitions that matter

These three words sound interchangeable but imply different trade-offs. Availability is the fraction of time the service accepts requests and returns non-error responses. Durability is whether committed data survives crashes, disk failures, and datacenter loss. Reliability is the probability the system performs correctly over a period — combining availability, durability, and correctness.

ShardPay's ledger must never lose a committed transfer (durability). The API should respond within 200ms for most requests (availability). Over a month, both should hold without reconciliation drift (reliability). You cannot maximize all three simultaneously during a partition — CAP forces a choice.

Durability before availability for payments

ShardPay will reject a write rather than acknowledge it before replication completes. During a network partition, the ledger chooses CP (consistency + partition tolerance): some clients see 503 Service Unavailable instead of a false success. A payment system that is "available" but loses transfers is worse than one that is briefly unavailable but correct.

This is the opposite of a social media feed, where showing a stale like count for 30 seconds is acceptable. ShardPay documents durability-first vs availability-first per component: ledger is CP, notification feed is AP, receipt PDF CDN is best-effort.

Example: AZ failure during peak load

US-East-1 loses power. ShardPay's ledger quorum loses one of three replicas but maintains majority in US-East-2. Writes continue with slightly higher latency. Availability dips to 99.97% for 20 minutes (RTO window). Durability holds: every transfer acknowledged before the failure is present on surviving replicas. No RPO breach. Customers see slower transfers, not lost money.

// ShardPay ledger write path — durability before client ack
public TransferResult commitTransfer(Transfer transfer) {
    // 1. Append to local WAL (durable on disk)
    long index = wal.append(transfer);

    // 2. Replicate to quorum before responding
    ReplicationResult result = raft.replicate(index, transfer);
    if (!result.quorumAcked()) {
        throw new UnavailableException("cannot achieve write quorum");  // CP: reject, don't lie
    }

    // 3. Only now tell the client success
    return TransferResult.committed(transfer.id(), index);
}

Measuring and budgeting nines

Nines are multiplicative across dependencies. If auth is 99.99% and ledger is 99.99%, the serial path is roughly 99.98% — you lose nines by chaining. ShardPay parallelizes fraud and ledger where possible and caches auth tokens to protect the product-level SLO.

Error budgets make nines actionable. At 99.99%, ShardPay gets ~52 minutes of downtime per year. That budget covers planned deploys (~15 min/month), unplanned failovers, and dependency blips. When the budget burns too fast, feature deploys pause and reliability work takes priority.

Reliability as the composite goal

Reliability is what customers actually experience: correct transfers, on time, every month. A system can be highly available (fast 503s) and highly durable (no data loss) but unreliable if it frequently returns wrong balances. ShardPay's reliability SLO includes a correctness dimension: reconciliation drift must stay below $0.01 per million dollars transferred.

Designing for reliability means choosing the right priority per component, measuring all three dimensions, and using error budgets to balance velocity against risk. Module 11 covers the observability stack that makes these metrics visible in production.