Skip to main content
PreparationZone Logo
PrepZone

Outbox and Inbox Patterns

Atomically write business state and an event — the bridge between DB commits and Kafka.

Pass checkpoint below

Why this matters

  • Dual write (commit ledger then publish to Kafka) loses messages if the broker call fails after DB commit — ShardPay lost settlement events in early architecture before outbox adoption.
  • Outbox is the standard pattern for reliable event publishing from transactional services.
  • Inbox dedup on consumers completes the story for at-least-once end-to-end effectively-once processing.
  • Interviewers ask how you atomically commit transfer row and event intent without 2PC across Postgres and Kafka.
Service
Business tableINSERT transfer
Outbox tableINSERT event
Message broker
Business row and outbox row commit in one DB transaction; relay publishes to broker.

The dual-write problem

Naive flow: BEGIN → insert transfer → COMMIT → kafka.send(). JVM dies after commit, before send — downstream never learns of transfer. Alternative: send first, then DB — broker has event DB lacks. Both orders fail one leg.

ShardPay needs atomicity between ledger state and event intent, not between two independent systems. The outbox table lives in the same Postgres database as transfers.

Walkthrough: crash after commit

Ledger transaction inserts transfer row and outbox row {eventId, payload, status=PENDING} in one commit. JVM crashes before relay runs. On restart relay polls WHERE status=PENDING, publishes to Kafka, marks SENT. No lost event — worst case delayed publish seconds until relay catches up.

Key points

  • Outbox table — append-only pending events in the same DB as business data, committed in one transaction. ShardPay's outbox_events holds Avro payload bytes, topic, partition key, and eventId for traceability.
  • Relay / CDC — polling publisher or Debezium CDC tail reads committed outbox rows and publishes to broker. ShardPay uses Debezium on outbox_events for sub-second relay latency; polling fallback in disaster mode.
  • Inbox table — consumer-side dedup store keyed by eventId; prevents duplicate apply on at-least-once redelivery. ShardPay pairs every outbox eventId with inbox unique constraint on projector services.
  • Dual-write problem — attempting separate commits to DB and broker without coordination; guaranteed partial failure under crash. Outbox eliminates dual write by making broker publish asynchronous and retryable after DB truth exists.
  • Ordering with outbox — relay publishes in created_at order per shard when using single relay partition; Debezium preserves binlog order. ShardPay includes sequence_num per aggregate for consumers that need strict per-transfer order beyond broker defaults.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • Inbox pattern on the consumer
  • Outbox operational concerns

Test yourself

Answer these before moving on — recall is what makes it stick.