Kafka Topic Partition

Question

Kafka Topic Partition offset position always start from 0 or random value and How to ensure the consumer record is the first record in the partition ? Is there any way to find out ? If any please let me know. Thanks.

Answer 1

Yes and no.

When you start a new topic, the offset start at zero. Depending on the Kafka version you are using, the offsets are

• logical – and incremented message by message (since 0.8.0: https://issues.apache.org/jira/browse/KAFKA-506) – or
• physical – ie, the offset is increased by the number of bytes for each message.

Furthermore, old log entries are cleared by configurable conditions:

• retention time: eg, keep message of the last week only
• retention size: eg, use at max 10GB of storage; delete old messages that cannot be stored any more
• log-compaction (since 0.8.1): you only preserve the latest value for each key (see https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction)

Thus, the first offset might not be zero if old messages got deleted. Furthermore, if you turn on log-compaction, some offsets might be missing.

In any case, you can always seek to any offset safely, as Kafka can figure out if the offset is valid or not. For an invalid offset, is automatically advances to the next valid offset. Thus, if you seek to offset zero, you will always get the oldest message that is stored.

Answer 2

Yes, Kafka offset starts from 0 and ends with byte length of the complete record and then next record picks the offset from there onward.

As Kafka is distributed so we can not assure that Consumer will get the data in ordered way.