Skip to main content
PreparationZone Logo
PrepZone

At-Most, At-Least, and Exactly-Once Delivery

What brokers promise vs what your handlers must enforce — ShardPay settlement events.

Pass checkpoint below

Why this matters

  • Brokers rarely deliver true exactly-once to external systems — your pipeline must name semantics per stage.
  • ShardPay settlement events use at-least-once Kafka delivery plus idempotent consumers; pretending the broker alone gives EOS caused duplicate settlement postings.
  • Duplicate settlement events without dedup double-credit merchant accounts — a production incident class, not a theoretical edge case.
  • Interviewers map at-most / at-least / exactly-once to concrete commit order and idempotency design.
At-most-onceMay lose messages
At-least-onceMay duplicate
Exactly-onceIdempotent pipeline
Exactly-once is a pipeline property — broker + producer + consumer must cooperate.

At-most-once

Producer fires without waiting for ack, or consumer commits offset before processing. Messages may be lost but never duplicated. ShardPay uses at-most-once only for non-critical telemetry (transfer.attempted metrics) where approximate counts suffice and loss is acceptable.

At-most-once is inappropriate for settlement or ledger projection topics — a lost TransferCompleted event means merchant balance never updates until manual reconciliation.

Walkthrough: metrics vs money

ShardPay's dashboard "transfers per minute" chart consumes at-most-once metrics from a UDP-sidecar — losing 0.1% of points does not affect money movement. The settlement projector consumes at-least-once with dedup — losing even one event is unacceptable.

Key points

  • At-most-once — a message is delivered zero or one times; loss is possible, duplication is not. ShardPay restricts this to metrics and debug taps, never to shardpay.settlement.posted.v1.
  • At-least-once — broker retries until ack; consumer may see duplicates after crash between process and commit. ShardPay's default for settlement pipelines: producer acks=all, consumer processes then commits offset, handler dedupes by eventId.
  • Exactly-once (EOS) — end-to-end effect as if each message processed once; requires broker transactions plus idempotent side effects on external stores. ShardPay uses Kafka EOS for producer→broker leg; Postgres writes still need idempotency keys because broker EOS does not span JDBC.
  • Idempotent consumer — processing the same message twice yields identical durable state. ShardPay's processed_events table with unique eventId makes settlement handler safe under redelivery.
  • Commit order — process side effects, mark dedup, then commit consumer offset — never commit before durable apply. Reversed order creates loss (at-most-once behavior) under crash.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • At-least-once — ShardPay's practical default
  • Exactly-once semantics end-to-end

Test yourself

Answer these before moving on — recall is what makes it stick.