Why this matters
- Dual write (commit ledger then publish to Kafka) loses messages if the broker call fails after DB commit — ShardPay lost settlement events in early architecture before outbox adoption.
- Outbox is the standard pattern for reliable event publishing from transactional services.
- Inbox dedup on consumers completes the story for at-least-once end-to-end effectively-once processing.
- Interviewers ask how you atomically commit transfer row and event intent without 2PC across Postgres and Kafka.
The dual-write problem
Naive flow: BEGIN → insert transfer → COMMIT → kafka.send(). JVM dies after commit, before send — downstream never learns of transfer. Alternative: send first, then DB — broker has event DB lacks. Both orders fail one leg.
ShardPay needs atomicity between ledger state and event intent, not between two independent systems. The outbox table lives in the same Postgres database as transfers.
Walkthrough: crash after commit
Ledger transaction inserts transfer row and outbox row {eventId, payload, status=PENDING} in one commit. JVM crashes before relay runs. On restart relay polls WHERE status=PENDING, publishes to Kafka, marks SENT. No lost event — worst case delayed publish seconds until relay catches up.
Key points
- Outbox table — append-only pending events in the same DB as business data, committed in one transaction. ShardPay's
outbox_eventsholds Avro payload bytes, topic, partition key, andeventIdfor traceability. - Relay / CDC — polling publisher or Debezium CDC tail reads committed outbox rows and publishes to broker. ShardPay uses Debezium on
outbox_eventsfor sub-second relay latency; polling fallback in disaster mode. - Inbox table — consumer-side dedup store keyed by
eventId; prevents duplicate apply on at-least-once redelivery. ShardPay pairs every outboxeventIdwith inbox unique constraint on projector services. - Dual-write problem — attempting separate commits to DB and broker without coordination; guaranteed partial failure under crash. Outbox eliminates dual write by making broker publish asynchronous and retryable after DB truth exists.
- Ordering with outbox — relay publishes in
created_atorder per shard when using single relay partition; Debezium preserves binlog order. ShardPay includessequence_numper aggregate for consumers that need strict per-transfer order beyond broker defaults.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Inbox pattern on the consumer
- Outbox operational concerns
Test yourself
Answer these before moving on — recall is what makes it stick.