Overcoming Message Duplication: Implementing Exactly-Once Semantics (EOS)

What the guarantee actually covers

Exactly-once processing in Kafka combines three mechanisms. Idempotent producers attach a producer id and sequence number so the broker discards duplicate writes from retries. Transactions let a producer write to multiple partitions atomically. Transactional offset commits place the consumer's offset commit inside that same transaction, so consuming, processing, and producing either all commit or all abort.

The result is a correct read-process-write loop entirely inside the streaming system. Consumers must set isolation level to read committed, or they will see aborted records and the guarantee evaporates at the read side.

Where the guarantee ends

The moment processing touches something outside the transaction — a payment API, an email send, an external database without a shared transaction — exactly-once no longer applies to that effect. The stream may replay, and the side effect will happen again. This boundary is the source of most production duplicate incidents attributed, incorrectly, to a broken EOS configuration.

The remedy at the boundary is idempotency, not stronger transactions. Give every message a stable business identifier, and make the external operation safe to repeat: a unique constraint on that identifier, an upsert, or an idempotency key honored by the downstream API.

Weigh the cost honestly

Transactions add latency because records become visible only at commit, and commit intervals directly set end-to-end delay. They add coordination overhead and complicate failure modes: a hung transaction blocks read-committed consumers until it times out. For high-volume telemetry where duplicates are harmless, at-least-once delivery with idempotent handlers is simpler, faster, and adequate.

A pragmatic rule: use transactions for stream-to-stream state changes with financial or contractual consequence, and use idempotent consumers everywhere effects leave the system.

Verify it rather than assume it

Test the guarantee with deliberate failure injection — kill consumers mid-transaction, force rebalances during commits, and partition the network between producer and broker. Then assert on the downstream state, counting business outcomes rather than messages. A duplicate-detection metric on the consuming side is worth maintaining permanently.