At-Most, At-Least, and Exactly-Once Delivery

What brokers promise vs what your handlers must enforce — ShardPay settlement events.

Advanced10 min read

Read these first

Chaos Engineering and Jepsen Testing

Why this matters

Brokers rarely deliver true exactly-once to external systems — your pipeline must name semantics per stage.
ShardPay settlement events use at-least-once Kafka delivery plus idempotent consumers; pretending the broker alone gives EOS caused duplicate settlement postings.
Duplicate settlement events without dedup double-credit merchant accounts — a production incident class, not a theoretical edge case.
Interviewers map at-most / at-least / exactly-once to concrete commit order and idempotency design.

At-most-onceMay lose messages

At-least-onceMay duplicate

Exactly-onceIdempotent pipeline

Exactly-once is a pipeline property — broker + producer + consumer must cooperate.

At-most-once

Producer fires without waiting for ack, or consumer commits offset before processing. Messages may be lost but never duplicated. ShardPay uses at-most-once only for non-critical telemetry (transfer.attempted metrics) where approximate counts suffice and loss is acceptable.

At-most-once is inappropriate for settlement or ledger projection topics — a lost TransferCompleted event means merchant balance never updates until manual reconciliation.

Walkthrough: metrics vs money

ShardPay's dashboard "transfers per minute" chart consumes at-most-once metrics from a UDP-sidecar — losing 0.1% of points does not affect money movement. The settlement projector consumes at-least-once with dedup — losing even one event is unacceptable.

Key points

At-most-once — a message is delivered zero or one times; loss is possible, duplication is not. ShardPay restricts this to metrics and debug taps, never to shardpay.settlement.posted.v1.
At-least-once — broker retries until ack; consumer may see duplicates after crash between process and commit. ShardPay's default for settlement pipelines: producer acks=all, consumer processes then commits offset, handler dedupes by eventId.
Exactly-once (EOS) — end-to-end effect as if each message processed once; requires broker transactions plus idempotent side effects on external stores. ShardPay uses Kafka EOS for producer→broker leg; Postgres writes still need idempotency keys because broker EOS does not span JDBC.
Idempotent consumer — processing the same message twice yields identical durable state. ShardPay's processed_events table with unique eventId makes settlement handler safe under redelivery.
Commit order — process side effects, mark dedup, then commit consumer offset — never commit before durable apply. Reversed order creates loss (at-most-once behavior) under crash.

At-least-once — ShardPay's practical default

Producer sends with acks=all and retries; broker appends to log. Consumer reads, applies settlement to merchant ledger projection, inserts eventId into inbox table, then commits Kafka offset. Crash after apply but before commit → redelivery → inbox dedup skips second apply.

This pipeline is at-least-once delivery with effectively-once processing — interviewers accept "at-least-once plus idempotency" as the production answer for payment side effects.

Walkthrough: consumer crash mid-handler

Settlement consumer processes eventId=e-4421, inserts merchant credit row, crashes before offset commit. Broker redelivers e-4421. Inbox lookup finds existing eventId; handler returns early; offset commits. Merchant credited once. Support sees one ledger line; Kafka lag clears.

Java

// ShardPay settlement consumer — at-least-once with idempotent apply (Java 17)
@KafkaListener(topics = "shardpay.settlement.posted.v1", groupId = "merchant-ledger-projector")
public void onSettlementPosted(ConsumerRecord<String, SettlementPosted> record) {
    String eventId = header(record, "eventId");
    if (inbox.alreadyProcessed(eventId)) {
        return; // safe on redelivery
    }

    settlementService.creditMerchant(record.value()); // upsert by transferId inside
    inbox.markProcessed(eventId, Duration.ofDays(30));
    // offset committed by container AFTER successful method return
}

Exactly-once semantics end-to-end

Kafka transactional producer plus read-process-write in same consumer transaction can achieve EOS within Kafka and compatible stores. ShardPay's payment ledger lives in Postgres outside the broker — true EOS requires outbox pattern, idempotent consumers, or transactional messaging bridges.

Name semantics honestly per stage: at-least-once on the wire, idempotent on the write, effectively-once for the business outcome.

Kafka trackSee delivery semantics in the Kafka track

Quick recall

Everything you need if you only revisit this box.

Exactly-once is an end-to-end pipeline property — broker EOS does not cover external Postgres.
At-least-once plus idempotent consumers is ShardPay's practical default for settlement.
Commit offset only after side effects and dedup marker are durable; name semantics per stage.

Test yourself

Answer these before moving on — recall is what makes it stick.