Kafka Consumer Groups Explained

A consumer group is a set of consumers that share the work of reading a topic. Kafka assigns each partition to exactly one member of the group, so adding consumers scales throughput up to the partition count.

One partition per consumer, at most

If a group has more consumers than partitions, the extras sit idle. Match your consumer count to your partition count for full utilization.

var props = new Properties();
props.put("group.id", "billing");
props.put("enable.auto.commit", "false");
 
var consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(List.of("orders"));

Commit after processing

Disable auto-commit and commit only once work succeeds, so a crash re-delivers rather than silently skips.

var records = consumer.poll(Duration.ofMillis(500));
for (var record : records) {
    handle(record);
}
consumer.commitSync();

Takeaways

• Different groups each get a full copy of the stream; members within a group split it.
• Rebalances pause consumption briefly — keep processing fast to avoid churn.
• Commit after success to get at-least-once delivery.

Tune max.poll.interval.ms so slow processing does not trigger a rebalance that kicks the consumer out of the group.