CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
gsc0441
gsc0441

Reputation: 323

how to get the all messages in a topic from kafka server

I would like to get all the messages from beginning in a topic from server.

Ex:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopic --from-beginning

When using the above console command, I would like to able to get all messages in a topic from the beginning but I couldn't consume all the messages in a topic from beginning using java code.

Upvotes: 15

Views: 81479

Answers (4)

gsc0441
gsc0441

Reputation: 323

TopicPartition topicPartition = new TopicPartition(topic, 0);
List<TopicPartition> partitions = Arrays.asList(topicPartition); 
consumer.assign(partitions);
consumer.seekToBeginning(partitions);

Upvotes: 8

KayV
KayV

Reputation: 13835

You can get all messages using the following command:

cd Users/kv/kafka/bin

./kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic topicName --from-beginning --max-messages 100

Upvotes: 12

Dinuka Hettiarachchi
Dinuka Hettiarachchi

Reputation: 77

Just change the consumer group

ConsumerConfig.GROUP_ID_CONFIG - to new group id

and set

AUTO_OFFSET_RESET_CONFIG - earliest

sample code- 

    props.put(ConsumerConfig.GROUP_ID_CONFIG, "newID");
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

Upvotes: 1

Manav Garg
Manav Garg

Reputation: 522

The easiest way would be to start a consumer and drain all the messages. Now I don't know how many partitions you have in your topic and whether you already have a an existing consumer group or not, but you have a few options:

Have a look at this API: https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

1) If you already have a consumer in the same consumer group, and still want to start consuming from the beginning, you should use the seek option listed in the API doc and set the offset to 0 for each consumer in the group. This would start consuming from the beginning.

2) Otherwise, you can start a few consumers in a new consumer group & you would not have to worry about seek.

PS: Please remember to provide more details about your setup in the future if you have more questions on Kafka. A lot of things depend on how you have configured your infrastructure & how you would prefer it to be and would thus vary from case to case.

Upvotes: 9

Related Questions