CODEWITHSUNDEEP
apache-kafka
Vaishu13
Vaishu13

Reputation: 455

kafka subscribe commit offset manually

I am using Kafka 9 and confused with the behavior of subscribe.

Upvotes: 1

Views: 3318

Answers (1)

Sönke Liebau
Sönke Liebau

Reputation: 1973

Why does it expects group.id with subscribe?

The concept of consumer groups is used by Kafka to enable parallel consumption of topics - every message will be delivered once per consumer group, no matter how many consumers actually are in that group. This is why the group parameter is mandatory, without a group Kafka would not know how this consumer should be treated in relation to other consumers that might subscribe to the same topic.

Whenever you start a consumer it will join a consumer group, based on how many other consumers are in this consumer group it will then be assigned partitions to read from. For these partitions it then checks whether a list read offset is known, if one is found it will start reading messages from this point. If no offset is found, the parameter auto.offset.reset controls whether reading starts at the earliest or latest message in the partition.

Do we need to commit the offset manually using commitSync? Even if don't do that I see that it always starts from the latest.

Whether or not you need to commit the offset depends on the value you choose for the parameter enable.auto.commit. By default this is set to true, which means the consumer will automatically commit its offset regularly (how often is defined by auto.commit.interval.ms). If you set this to false, then you will need to commit the offsets yourself. This default behavior is probably also what is causing your "problem" where your consumer always starts with the latest message. Since the offset was auto-committed it will use that offset.

Is there a way a replay the messages from beginning?

If you want to start reading from the beginning every time, you can call seekToBeginning, which will reset to the first message in all subscribed partitions if called without parameters, or just those partitions that you pass in.

Upvotes: 3

Related Questions