Kafka Fundamentals: Topics, Partitions, and Offsets

Kafka is a distributed, append-only log. Messages go into topics, topics split into partitions for parallelism, and each message gets a monotonic offset within its partition. Those three concepts explain almost everything else.

Partitions are the unit of parallelism

Order is guaranteed only within a partition. The key you choose decides which partition a message lands in, so related events stay ordered.

// same key -> same partition -> ordered per customer
var record = new ProducerRecord<>("orders", customerId, orderJson);
producer.send(record);

Offsets track progress

Consumers commit the offset they have processed. On restart they resume from the last committed position, not the beginning.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group billing

Takeaways

• Choose partition keys so ordering-sensitive events share a partition.
• More partitions means more parallelism but more overhead — size deliberately.
• Retention is time- or size-based; Kafka keeps data even after it is read.

Partition count is hard to reduce later, so start with a number that gives room to scale consumers without over-provisioning.