CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-apikafka-producer-api
vbNewbie
vbNewbie

Reputation: 3345

creating/updating topic in kafka -.81 with partitions

I would like to create a topic in kafka using version 0.81 and do not have the kafka-topics.sh script at my disposal to do the command line create command. I need to ensure the topic has multiple partitions in order for multithreaded consumers to consume messages from the topic. Each message should only be delivered once and I was told this cannot be done without partitions in a queue even if using the same group ID.

My other question is whether a single consumer run as two separate processes can do the same thing using the same group ID? Basically I am currently not reading the messages from the queue fast enough.

Because of the older version I am using I cannot update the topic with partition count as suggested by this command:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic 
   my-topic --replication-factor 1 --partitions 2

Upvotes: 1

Views: 676

Answers (1)

Seweryn Habdank-Wojewódzki
Seweryn Habdank-Wojewódzki

Reputation: 2177

My other question is whether a single consumer run as two separate processes can do the same thing using the same group ID? Basically I am currently not reading the messages from the queue fast enough.

If I understand your question, the answer is YES. I mean, group.id works in the following way.

  1. If it is the same, it will make, that consumers within group.id will work as with load balancer
  2. If group.id is different, every group will got own "copy" of the message, so you can perform different operations on the same message content.

Because of the older version I am using I cannot update the topic with partition count as suggested by this command

I am pretty confused what are you doing here, but to change parameters of the topic, you shall use "--alter" opion/command and not "--create".

Upvotes: 1

Related Questions