Skip to main content
PreparationZone Logo
PrepZone

Compensation and Eventual Consistency

Design reversible operations and reconcile ledgers when perfect atomicity is impossible.

Pass checkpoint below

Why this matters

  • Every distributed payment platform has stuck sagas, duplicate events, and ledger drift — perfection is not the goal; detectable convergence is.
  • Compensation is a business operation with audit requirements, not a silent database rollback.
  • ShardPay runs nightly reconciliation against bank settlement files and alerts on any imbalance > $0.01.
  • Senior interviews expect you to describe what happens at 2 a.m. when a saga has been DEBITED for 45 minutes with no credit.
Orchestrator
Debit shard
Credit shard
Compensate debit
Central coordinator executes steps and triggers compensations on failure.

The consistency gap

In a saga, intermediate states are visible. After debit succeeds but before credit completes, ShardPay's ledger shows money removed from account A without a matching entry on account B — temporarily inconsistent from a global view. This is by design, not a bug, as long as the gap is bounded and monitored.

Merchant dashboards may briefly show a lower balance on the source account. ShardPay displays "transfer in progress" status tied to transferId so merchants understand the transient state rather than assuming theft.

Key points

  • Intermediate inconsistency — valid transient state between saga steps. ShardPay tolerates up to 30 seconds of cross-shard imbalance per transfer; beyond that, step timeouts trigger compensation.
  • Convergence — the system reaches a correct final state via forward completion, compensation, or reconciliation. ShardPay's target: 99.99% of transfers converge within 60 seconds without human intervention.
  • Compensating transaction — a new ledger entry that semantically undoes a prior step. CompensateDebit on shard 3 creates a credit with reason code SAGA_ROLLBACK, linked to the original debit by reference ID.
  • Reconciliation — batch process comparing internal ledger to external source of truth. ShardPay reconciles against ACH settlement files nightly and against card network auth captures hourly.

Compensation in practice

Compensation must be idempotent, auditable, and safe to retry. ShardPay never deletes a debit row — it posts an offsetting credit with full provenance.

Walkthrough: stuck saga detected by reconciliation

  1. Transfer TXN-7721 debited $800 from merchant A on shard 2 at 14:03 UTC.
  2. Credit to merchant B on shard 5 failed silently due to a misconfigured timeout — saga stuck in DEBITED.
  3. At 14:45, step timeout should have triggered compensation but a bug skipped the alert (fixed in postmortem).
  4. At 02:00 UTC, nightly reconciliation finds $800 imbalance: debit without matching credit or compensation.
  5. Reconciliation job idempotently runs CompensateDebit for TXN-7721, marks saga RECONCILED, pages on-call for root-cause review.
Java
// ShardPay reconciliation — detect and compensate stuck debits
@Scheduled(cron = "0 0 2 * * *") // nightly 02:00 UTC
public void reconcileStuckTransfers() {
    List<StuckTransfer> stuck = sagaRepo.findDebitedOlderThan(Duration.ofHours(1));
    for (StuckTransfer t : stuck) {
        if (reconciliationLog.alreadyHandled(t.transferId())) continue;

        if (t.hasMatchingCredit()) {
            sagaRepo.markCompleted(t.sagaId()); // forward path completed, state lag
        } else {
            shardClient.compensateDebit(t.fromShard(), t.debitRef());
            sagaRepo.markReconciled(t.sagaId(), "AUTO_COMPENSATE");
        }
        reconciliationLog.record(t.transferId(), Instant.now());
    }
}

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 1 section

  • Reconciliation layers

Test yourself

Answer these before moving on — recall is what makes it stick.