Tuning Kafka Producer Throughput

The Kafka producer is fast out of the box but conservative. A handful of settings trade a little latency for a large gain in throughput when you are writing high volumes.

Batch and linger

batch.size and linger.ms let the producer group records into fewer, larger requests. A few milliseconds of linger dramatically improves batching.

props.put("batch.size", 64 * 1024);
props.put("linger.ms", 10);
props.put("compression.type", "lz4");
props.put("acks", "all");

Balance durability and speed

acks=all is safest but slowest; acks=1 is faster with a small risk window. Choose based on how much data loss you can tolerate.

props.put("acks", "1");   // faster, tolerates rare loss on leader failure

Takeaways

• Compression shrinks network and disk usage; lz4 and zstd are good defaults.
• Larger batches raise throughput but add latency — measure your workload.
• Keep enable.idempotence=true to avoid duplicates even while tuning.

Watch buffer.memory; if the producer buffer fills, send blocks, so size it for your peak in-flight volume.