Exactly-Once in Kafka: The Idempotent Producer, Transactions, read_committed, and the Boundary Everyone Crosses

A practical guide to the guarantee almost everyone thinks they have and very few actually do: the difference between delivering a message once and applying its effect once, the two duplicates that are not the same problem (producer-side retries and consumer-side rebalances), the idempotent producer from the inside with PID, epoch and per-partition sequence numbers — plus the three settings it forces, including the five-in-flight limit and why it exists — transactions with a stable transactional.id, zombie fencing in initTransactions, and sendOffsetsToTransaction as the piece that makes the guarantee real, the complete consume-transform-produce loop in Java and in Python with confluent-kafka, the three details nearly every implementation gets wrong (the seek after an abort, which is silent data loss; the +1 on the offset; and the groupMetadata overload that enables EOS v2), read_committed and the LSO with the offsets that appear to go missing and the aborted messages filtered client-side, Kafka Streams with exactly_once_v2 and the commit.interval.ms almost nobody touches, KIP-890 and transaction.version=2 in Kafka 4.0 with the per-commit epoch bump that closes the cross-transaction message leak, and the boundary that voids all of it the moment you call a payment gateway. With eight recurring mistakes, the hanging-transaction runbook using kafka-transactions.sh, the alert that actually works (the log-end-offset to LSO gap, not lag), the real cost in throughput and latency, a production checklist, FAQ and glossary. Production-ready code in Java, Python and shell. September 20, 2026 expansion: the internals with PID, epoch and per-partition sequence numbers, why the in-flight limit is five, and the UNKNOWN_PRODUCER_ID that looks alarming without being a problem; the transaction coordinator and the __transaction_state log step by step, with PrepareCommit as the point of no return and why a hanging transaction blocks consumers that are not yours; the transactional.id as the decision almost nobody gets right, with the StatefulSet that provides stable identifiers and a closing answer to the historical KIP-447 and groupMetadata() question; what read_committed does to your latency, covering the LSO, transaction size and the aborted-transaction index you pay for in bandwidth; Kafka Connect with exactly.once.source.support and its two-phase rolling restart, and why there is no equivalent switch for sink connectors; Flink with transactional-id-prefix and the timing trap that causes silent data loss; the three patterns for when the guarantee leaves Kafka (inbox, outbox and idempotent writes) with ready-to-use SQL; multi-cluster and why MirrorMaker 2 does not carry the guarantee across; five metrics and two Prometheus alerts; fault-injection tests with Testcontainers and the difference between fatal and retryable errors; a complete case where latency rises 29% and incidents drop to zero; when not to use exactly-once; how to enable transaction.version=2 without surprises; and four questions that always come up afterwards.