Why this matters
- Enterprise interviews still probe XA transactions and JDBC
commit()across datasources. - ShardPay experimented with XA for cross-shard transfers in 2023 and rolled it back after a coordinator outage left 400 accounts locked for 11 minutes.
- Understanding 2PC limits explains why sagas, outbox patterns, and eventual consistency replaced distributed transactions in payment systems.
- Every senior design review should answer: "Why not 2PC here?" with concrete failure modes, not hand-waving.
How 2PC works
A coordinator runs two phases. In phase 1 (prepare), each participant locks resources and votes yes or no. In phase 2 (commit or abort), the coordinator broadcasts the decision; participants apply it and release locks.
ShardPay's abandoned XA flow looked like this: the transfer API opened an XA transaction, prepared a debit on shard A and a credit on shard B, then committed both or aborted both. On paper, balances stayed consistent. In production, the prepare phase held row locks on both shards until commit completed — contending with normal merchant traffic.
Prepare phase in ShardPay's XA experiment
When merchant M-4421 initiated a $500 cross-shard transfer, shard A's participant locked the source row and wrote a prepare log entry. Shard B did the same for the destination. Neither balance changed yet, but both rows were unavailable to other transfers targeting those accounts. Under peak load, lock wait times on hot merchant accounts exceeded 800ms — unacceptable for ShardPay's 200ms transfer SLO.
Key points
- Coordinator — the single process that collects votes and issues commit or abort. In ShardPay's XA experiment, the transfer service JVM acted as coordinator; when it GC-paused for 9 seconds, all prepared shards blocked.
- Prepare vote — each participant locks resources and responds yes/no without making the change permanent. ShardPay shard participants returned yes only after verifying sufficient balance and acquiring row locks.
- Commit log — participants write the decision durably before acknowledging. If a participant crashes after commit but before unlock, recovery replays the log — but only if the coordinator's decision is known.
- In-doubt transaction — a participant voted yes but never received commit or abort. ShardPay's incident left 400 in-doubt transactions until the coordinator's recovery timeout fired at 11 minutes.
Why 2PC breaks in production
2PC optimizes for atomicity, not availability. Three failure modes dominate real systems: coordinator crash, participant crash after prepare, and network partition between coordinator and participants.
Coordinator failure blocks everyone
If the coordinator dies after all participants vote yes, every participant holds locks and waits. JDBC XA defaults to a timeout (often minutes), during which prepared rows reject new transactions. ShardPay's 2023 outage matched this exactly: a pod OOM killed the coordinator mid-2PC; shards A and B held locks until XA recovery timed out.
Partitions create impossible choices
During a network split, some participants may receive commit while others never hear from the coordinator. Without a separate consensus layer, 2PC cannot guarantee both consistency and progress under partition — it is not partition-tolerant for cross-region deployments. ShardPay runs shards in three AZs within one region; even there, transient network blips caused prepare timeouts that aborted transfers merchants had already seen as "processing."
// ShardPay XA transfer — simplified; removed from production in 2023
@Transactional
public TransferResult crossShardTransfer(TransferRequest req) {
// Phase 1: XA prepare on both datasources (locks held)
xaDebitShard.prepare(req.fromAccount(), req.amountCents());
xaCreditShard.prepare(req.toAccount(), req.amountCents());
// Phase 2: commit both — if coordinator dies here, shards block
xaDebitShard.commit();
xaCreditShard.commit();
return TransferResult.completed(req.transferId());
}
Alternatives ShardPay adopted
After retiring XA, ShardPay uses saga orchestration for cross-shard transfers and accepts brief intermediate states visible to reconciliation. The trade-off is explicit: availability and partition tolerance over cross-shard atomicity.
Saga vs 2PC at a glance
| Property | 2PC | Saga (orchestrated) |
|---|---|---|
| Atomicity | All-or-nothing at commit instant | Eventual; compensations on failure |
| Blocking | Yes, on coordinator failure | No; each step commits independently |
| Partition tolerance | Poor cross-region | Good with idempotent steps |
| Lock duration | Held until phase 2 | Per-step, short-lived |
Atomicity
2PCAll-or-nothing at commit instantSaga (orchestrated)Eventual; compensations on failureBlocking
2PCYes, on coordinator failureSaga (orchestrated)No; each step commits independentlyPartition tolerance
2PCPoor cross-regionSaga (orchestrated)Good with idempotent stepsLock duration
2PCHeld until phase 2Saga (orchestrated)Per-step, short-lived
2PC vs saga comparison for ShardPay
ShardPay's current flow debits shard A, credits shard B, then marks complete — with a compensating credit-back if step 2 fails. Reconciliation catches any stuck state within 15 minutes.
Key points
- Blocking protocol — participants cannot progress without coordinator instructions after voting yes. ShardPay measured p99 transfer latency doubling during XA lock contention on top accounts.
- Single coordinator SPOF — unless the coordinator itself runs on Raft/Paxos, it is a blast-radius concentrator. ShardPay now distributes saga state in a dedicated orchestrator DB with leader election.
- Heuristic rollback — when timeout fires, participants guess abort; if the coordinator actually committed, data diverges. ShardPay's XA logs showed two heuristic aborts in staging that would have caused silent balance drift in prod.
- Latency tax — two round-trips minimum (prepare + commit) plus lock hold time. ShardPay's saga completes in one round-trip per shard with no cross-shard lock coupling.
Quick recall
Everything you need if you only revisit this box.
- 2PC provides atomicity but blocks participants when the coordinator fails.
- Cross-region 2PC is not partition-tolerant — sagas are the modern default.
- ShardPay retired XA after lock contention and coordinator SPOF incidents.
Test yourself
Answer these before moving on — recall is what makes it stick.