Compensation and Eventual Consistency

Design reversible operations and reconcile ledgers when perfect atomicity is impossible.

Advanced10 min read

Read these first

Saga Pattern — Choreography

Why this matters

Every distributed payment platform has stuck sagas, duplicate events, and ledger drift — perfection is not the goal; detectable convergence is.
Compensation is a business operation with audit requirements, not a silent database rollback.
ShardPay runs nightly reconciliation against bank settlement files and alerts on any imbalance > $0.01.
Senior interviews expect you to describe what happens at 2 a.m. when a saga has been DEBITED for 45 minutes with no credit.

Orchestrator

Debit shard

Credit shard

Compensate debit

Central coordinator executes steps and triggers compensations on failure.

The consistency gap

In a saga, intermediate states are visible. After debit succeeds but before credit completes, ShardPay's ledger shows money removed from account A without a matching entry on account B — temporarily inconsistent from a global view. This is by design, not a bug, as long as the gap is bounded and monitored.

Merchant dashboards may briefly show a lower balance on the source account. ShardPay displays "transfer in progress" status tied to transferId so merchants understand the transient state rather than assuming theft.

Key points

Intermediate inconsistency — valid transient state between saga steps. ShardPay tolerates up to 30 seconds of cross-shard imbalance per transfer; beyond that, step timeouts trigger compensation.
Convergence — the system reaches a correct final state via forward completion, compensation, or reconciliation. ShardPay's target: 99.99% of transfers converge within 60 seconds without human intervention.
Compensating transaction — a new ledger entry that semantically undoes a prior step. CompensateDebit on shard 3 creates a credit with reason code SAGA_ROLLBACK, linked to the original debit by reference ID.
Reconciliation — batch process comparing internal ledger to external source of truth. ShardPay reconciles against ACH settlement files nightly and against card network auth captures hourly.

Compensation in practice

Compensation must be idempotent, auditable, and safe to retry. ShardPay never deletes a debit row — it posts an offsetting credit with full provenance.

Walkthrough: stuck saga detected by reconciliation

Transfer TXN-7721 debited $800 from merchant A on shard 2 at 14:03 UTC.
Credit to merchant B on shard 5 failed silently due to a misconfigured timeout — saga stuck in DEBITED.
At 14:45, step timeout should have triggered compensation but a bug skipped the alert (fixed in postmortem).
At 02:00 UTC, nightly reconciliation finds $800 imbalance: debit without matching credit or compensation.
Reconciliation job idempotently runs CompensateDebit for TXN-7721, marks saga RECONCILED, pages on-call for root-cause review.

Java

// ShardPay reconciliation — detect and compensate stuck debits
@Scheduled(cron = "0 0 2 * * *") // nightly 02:00 UTC
public void reconcileStuckTransfers() {
    List<StuckTransfer> stuck = sagaRepo.findDebitedOlderThan(Duration.ofHours(1));
    for (StuckTransfer t : stuck) {
        if (reconciliationLog.alreadyHandled(t.transferId())) continue;

        if (t.hasMatchingCredit()) {
            sagaRepo.markCompleted(t.sagaId()); // forward path completed, state lag
        } else {
            shardClient.compensateDebit(t.fromShard(), t.debitRef());
            sagaRepo.markReconciled(t.sagaId(), "AUTO_COMPENSATE");
        }
        reconciliationLog.record(t.transferId(), Instant.now());
    }
}

Reconciliation layers

ShardPay runs reconciliation at three horizons: real-time (step timeouts), hourly (network settlement), and nightly (full ledger vs bank statements).

Real-time: saga age monitoring

Metrics track saga_age_seconds by state. Alert fires when p99 age of DEBITED sagas exceeds 5 minutes. On-call runbook: check orchestrator health, shard connectivity, then manual compensate if auto-compensation failed.

Hourly: payment network alignment

Card captures and ACH batches arrive with a delay. ShardPay compares internal SETTLED events against network files. Mismatches create tickets in the ops queue — not auto-compensated because the network file may be incomplete.

Nightly: bank statement truth

The finance team's bank statement is the ultimate source of truth for fiat movement. ShardPay's ledger must sum to the statement within tolerance. Drift > $0.01 per merchant triggers investigation; systematic drift triggers a severity-1 incident.

Key points

Idempotent reconciliation — safe to rerun without double-compensating. ShardPay's reconciliation_log table stores (transferId, action, timestamp) with unique constraint on transferId.
Human-in-the-loop — auto-compensation handles known patterns; ambiguous cases escalate. ShardPay requires manual approval for compensation > $10,000 or involving flagged fraud accounts.
Audit trail — every compensation links to saga ID, reason code, and triggering system (timeout, reconciliation, manual). Regulators ask for this chain during examinations.
Merchant communication — compensating a debit means telling the merchant why their transfer failed. ShardPay's API returns status: FAILED, reason: CREDIT_UNAVAILABLE with webhook notification.

Quick recall

Everything you need if you only revisit this box.

Intermediate inconsistency is expected — design for bounded convergence time.
Compensation is an auditable business transaction, not a row delete.
Reconciliation at multiple horizons catches what sagas miss.

Test yourself

Answer these before moving on — recall is what makes it stick.