CODEWITHSUNDEEP
apache-kafka
SQL.injection
SQL.injection

Reputation: 2647

does kakfka-console-consumer consumes from multiple partitions

I have a kafka topic with multiple partitions.

I want to dump the topic into a file to conduct some analysis there, therefore i think the easiest way to do it is to use the kafka-console-consumer.

My question is, does the kafka-console-consumer is able to read to all the topic partitions, or it will be assigned to a single partition?

If not, how do assign the kafka-console-consumer to a specific partition, therefore a would have to start as many kafka-console-consumers as partitions.

Upvotes: 0

Views: 2273

Answers (2)

Edenhill
Edenhill

Reputation: 3113

As previously mentioned by Lukas, you can use kafkacat to dump all messages from a topic to a local file:

kafkacat -b a_broker -t your_topic -o beginning > all_msgs.bin

This will consume messages from all partitions, if you want to limit to a single partition use the -p <partition> switch.

The default output message separator is newline (\n), if you want something else use the -f "<fmt>" option.

Upvotes: 1

Luk&#225;š Havrlant
Luk&#225;š Havrlant

Reputation: 4414

Yes, kafka-console-consumer reads from all available partitions. Also take a look at kafkacat, it has some more advanced but usefull features.

Upvotes: 3

Related Questions