Why this matters
- Brokers rarely deliver true exactly-once to external systems — your pipeline must name semantics per stage.
- ShardPay settlement events use at-least-once Kafka delivery plus idempotent consumers; pretending the broker alone gives EOS caused duplicate settlement postings.
- Duplicate settlement events without dedup double-credit merchant accounts — a production incident class, not a theoretical edge case.
- Interviewers map at-most / at-least / exactly-once to concrete commit order and idempotency design.
At-most-once
Producer fires without waiting for ack, or consumer commits offset before processing. Messages may be lost but never duplicated. ShardPay uses at-most-once only for non-critical telemetry (transfer.attempted metrics) where approximate counts suffice and loss is acceptable.
At-most-once is inappropriate for settlement or ledger projection topics — a lost TransferCompleted event means merchant balance never updates until manual reconciliation.
Walkthrough: metrics vs money
ShardPay's dashboard "transfers per minute" chart consumes at-most-once metrics from a UDP-sidecar — losing 0.1% of points does not affect money movement. The settlement projector consumes at-least-once with dedup — losing even one event is unacceptable.
Key points
- At-most-once — a message is delivered zero or one times; loss is possible, duplication is not. ShardPay restricts this to metrics and debug taps, never to
shardpay.settlement.posted.v1. - At-least-once — broker retries until ack; consumer may see duplicates after crash between process and commit. ShardPay's default for settlement pipelines: producer
acks=all, consumer processes then commits offset, handler dedupes byeventId. - Exactly-once (EOS) — end-to-end effect as if each message processed once; requires broker transactions plus idempotent side effects on external stores. ShardPay uses Kafka EOS for producer→broker leg; Postgres writes still need idempotency keys because broker EOS does not span JDBC.
- Idempotent consumer — processing the same message twice yields identical durable state. ShardPay's
processed_eventstable with uniqueeventIdmakes settlement handler safe under redelivery. - Commit order — process side effects, mark dedup, then commit consumer offset — never commit before durable apply. Reversed order creates loss (at-most-once behavior) under crash.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- At-least-once — ShardPay's practical default
- Exactly-once semantics end-to-end
Test yourself
Answer these before moving on — recall is what makes it stick.