CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Greg Bala
Greg Bala

Reputation: 3903

How to read the entire Kafka topic, from beginning to end, without changing the group.id and without knowing about topic's partition

I am reading from Kafka topic, and I commit the read offsets.

Can I somehow, using the same group.id and without seeking on a particular partition as suggested here, re-read the entire topic from beginning to end?

using Kafka 1.1 my pseudo code:

var config = new Dictionary<string, object>
{
 { "group.id", "NCC1"},
 ...
 { "enable.auto.commit", "false"},
 { "default.topic.config", new Dictionary<string, object>
 {
  { "auto.offset.reset", "earliest" } 
 }
 }
};      

using (var consumer = new Consumer<string, string>(config, new StringDeserializer(Encoding.UTF8), new StringDeserializer(Encoding.UTF8)))
{
  consumer.Subscribe(new string[] { topic });
  consumer.OnMessage += (_, msg) =>
  {
    // DO SOMETHING
    onsumer.CommitAsync(msg);
  };

Upvotes: 2

Views: 2542

Answers (2)

Nishu Tayal
Nishu Tayal

Reputation: 20880

You can use kafka-consumer-group to reset offsets for the existing group ids.

kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> 
--reset-offsets --to-earliest --execute

It will reset the offset for the given topic and group.id to the earliest.

You can see more options there:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling

Upvotes: 2

HL&#39;REB
HL&#39;REB

Reputation: 876

Kafka allows you to reset the offset for a particular topic and group id. This possibility of replay is actually one of the selling points of Kafka.

You can do it using the kafka-streams-application-reset commandline tool as documented here.

An alternative way is to use kafka-consumer-groups command line tool as described in this blog. The alternative way might be interesting if you are using Windows since the equivalent batch file for kafka-streams-application-reset.sh is not provided.

Obviously, you can move your offset backward as long as the events are still in the topic. Events already removed from the topic cannot be replayed.

EDIT

I think your question is quite similar to this one

Upvotes: 1

Related Questions