Saga Pattern — Orchestration

Orchestrated cross-shard transfer

ShardPay's transfer saga has four steps: validate fraud score, debit source shard, credit destination shard, emit settlement event. The orchestrator persists state after each step so a crash mid-saga resumes from the last completed step, not from scratch.

When merchant M-8812 sends $1,200 from account A-US-7 (shard 3) to A-EU-2 (shard 9), the orchestrator creates saga instance SAGA-9f3a in state STARTED, calls fraud service, then transitions to DEBIT_PENDING. Only after a successful debit does it attempt credit — never both in one atomic operation.

Saga state machine

ShardPay models each transfer as an explicit state machine. Illegal transitions are rejected at the orchestrator — participants cannot skip ahead or commit out of order.

States and transitions

STARTED → FRAUD_OK → DEBITED → CREDITED → COMPLETED
                ↓         ↓         ↓
             ABORTED  COMPENSATING → COMPENSATED

A failure at CREDITED is rare (credit succeeded but emit failed); the orchestrator retries the emit step idempotently rather than compensating a successful credit. Compensation is reserved for steps that must be undone — debiting without a matching credit.

// ShardPay orchestrated saga — core state transitions
public void advance(SagaInstance saga) {
    switch (saga.state()) {
        case STARTED -> {
            fraudService.check(saga.transferId());
            saga.transition(FRAUD_OK);
            persist(saga);
            advance(saga); // tail-recurse to next step
        }
        case FRAUD_OK -> {
            DebitResult debit = shardClient.debit(saga.fromShard(), saga.request());
            saga.recordDebit(debit);
            saga.transition(DEBITED);
            persist(saga);
            advance(saga);
        }
        case DEBITED -> {
            try {
                shardClient.credit(saga.toShard(), saga.request());
                saga.transition(CREDITED);
                persist(saga);
            } catch (CreditException e) {
                compensateDebit(saga);
            }
        }
        case CREDITED -> {
            outbox.publish(SettlementEvent.from(saga));
            saga.transition(COMPLETED);
            persist(saga);
        }
        default -> { /* terminal or compensating — no-op */ }
    }
}

private void compensateDebit(SagaInstance saga) {
    saga.transition(COMPENSATING);
    persist(saga);
    shardClient.creditBack(saga.fromShard(), saga.debitRef());
    saga.transition(COMPENSATED);
    persist(saga);
}

Recovery and idempotency

Orchestrator crashes are expected. On restart, ShardPay scans sagas in non-terminal states and resumes from the last persisted step. Each participant call carries sagaId + stepName as an idempotency key so retries do not double-debit.

Walkthrough: orchestrator crash after debit

1. Saga SAGA-9f3a reaches state DEBITED; debit of $1,200 confirmed on shard 3.
2. Orchestrator pod killed by K8s node drain before credit attempt.
3. New orchestrator leader loads SAGA-9f3a, sees DEBITED, retries credit with idempotency key SAGA-9f3a:CREDIT.
4. Shard 9 applies credit once; saga moves to CREDITED, then COMPLETED.

If shard 9 already received the credit (orchestrator crashed after credit but before persist), the idempotency key returns the stored result and the saga advances without duplicate credit.