Exactly-Once Semantics in Kafka

By default Kafka gives at-least-once delivery: a retried send can produce a duplicate. For consume-transform-produce pipelines, transactions give you end-to-end exactly-once semantics.

Idempotent producer

Enabling idempotence deduplicates retries within a partition automatically.

props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("transactional.id", "payments-tx-1");

Wrap read-process-write in a transaction

Committing offsets inside the same transaction as the output makes the whole step atomic.

producer.initTransactions();
producer.beginTransaction();
producer.send(outputRecord);
producer.sendOffsetsToTransaction(offsets, groupMetadata);
producer.commitTransaction();

Takeaways

• Set consumers to isolation.level=read_committed to skip aborted messages.
• Exactly-once adds latency and coordination cost — use it only where required.
• For side effects outside Kafka, you still need idempotent writes downstream.

Exactly-once covers Kafka-to-Kafka flows; a database write in the middle still needs its own idempotency key to avoid double application.