Why this matters
- Every distributed payment platform has stuck sagas, duplicate events, and ledger drift — perfection is not the goal; detectable convergence is.
- Compensation is a business operation with audit requirements, not a silent database rollback.
- ShardPay runs nightly reconciliation against bank settlement files and alerts on any imbalance > $0.01.
- Senior interviews expect you to describe what happens at 2 a.m. when a saga has been
DEBITEDfor 45 minutes with no credit.
The consistency gap
In a saga, intermediate states are visible. After debit succeeds but before credit completes, ShardPay's ledger shows money removed from account A without a matching entry on account B — temporarily inconsistent from a global view. This is by design, not a bug, as long as the gap is bounded and monitored.
Merchant dashboards may briefly show a lower balance on the source account. ShardPay displays "transfer in progress" status tied to transferId so merchants understand the transient state rather than assuming theft.
Key points
- Intermediate inconsistency — valid transient state between saga steps. ShardPay tolerates up to 30 seconds of cross-shard imbalance per transfer; beyond that, step timeouts trigger compensation.
- Convergence — the system reaches a correct final state via forward completion, compensation, or reconciliation. ShardPay's target: 99.99% of transfers converge within 60 seconds without human intervention.
- Compensating transaction — a new ledger entry that semantically undoes a prior step.
CompensateDebiton shard 3 creates a credit with reason codeSAGA_ROLLBACK, linked to the original debit by reference ID. - Reconciliation — batch process comparing internal ledger to external source of truth. ShardPay reconciles against ACH settlement files nightly and against card network auth captures hourly.
Compensation in practice
Compensation must be idempotent, auditable, and safe to retry. ShardPay never deletes a debit row — it posts an offsetting credit with full provenance.
Walkthrough: stuck saga detected by reconciliation
- Transfer
TXN-7721debited $800 from merchant A on shard 2 at 14:03 UTC. - Credit to merchant B on shard 5 failed silently due to a misconfigured timeout — saga stuck in
DEBITED. - At 14:45, step timeout should have triggered compensation but a bug skipped the alert (fixed in postmortem).
- At 02:00 UTC, nightly reconciliation finds $800 imbalance: debit without matching credit or compensation.
- Reconciliation job idempotently runs
CompensateDebitforTXN-7721, marks sagaRECONCILED, pages on-call for root-cause review.
// ShardPay reconciliation — detect and compensate stuck debits
@Scheduled(cron = "0 0 2 * * *") // nightly 02:00 UTC
public void reconcileStuckTransfers() {
List<StuckTransfer> stuck = sagaRepo.findDebitedOlderThan(Duration.ofHours(1));
for (StuckTransfer t : stuck) {
if (reconciliationLog.alreadyHandled(t.transferId())) continue;
if (t.hasMatchingCredit()) {
sagaRepo.markCompleted(t.sagaId()); // forward path completed, state lag
} else {
shardClient.compensateDebit(t.fromShard(), t.debitRef());
sagaRepo.markReconciled(t.sagaId(), "AUTO_COMPENSATE");
}
reconciliationLog.record(t.transferId(), Instant.now());
}
}
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 1 section
- Reconciliation layers
Test yourself
Answer these before moving on — recall is what makes it stick.