CODEWITHSUNDEEP
apache-kafka
Kai Chen
Kai Chen

Reputation: 153

Create multiple consumers in Kafka in command line

I'm new in Kafka. When I was running the quick start example in command line, I found I can't create multiple consumers in command line.

Condition:

I built a topic named test with 3 partitions, and I also built a producer on this topic.

Then I wanted to create two different consumers sharing a same consumer-group named test1 on this topic.

I ran the command like below twice:

   bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --group test1

The first one worked but when I ran the second time the first one would disconnect and the second one worked.

So how can I create two or more consumers in a same consumer group in command line?

    WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)

Upvotes: 15

Views: 48084

Answers (5)

Rudy
Rudy

Reputation: 481

When you consume Topic without groupid Kafka create random groupid for your session. You can specify groupid --consumer-property group.id=test-consumer-group if groupid exist or you can add to your session new groupid(name) when you consume if group not exist --topic second-topic --group my-first-group p and Kafka will create new group

Upvotes: 0

Tapan Kumar
Tapan Kumar

Reputation: 9

use this:

--partition <Integer: partition>        The partition to consume from.

Upvotes: 1

Joey
Joey

Reputation: 339

Besides using --consumer.config option like the secfree's answer, you can also use

--consumer-property group.id=your_group

option to specify a group name without editing the config file.

Upvotes: 21

Ashwini Kumar
Ashwini Kumar

Reputation: 299

You can use the below command to create the consumers in the group "test-consumer-group" to "test" topic:

bin/kafka-console-consumer.sh --bootstrap-server <brokerIP>:9092 --topic test --consumer-property group.id=test-consumer-group

Below command will list the consumer group configuration:

bin/kafka-consumer-groups.sh --bootstrap-server <brokerIP>:9092 --describe --group test-consumer-group

Eg:

GROUP || TOPIC || PARTITION || CURRENT-OFFSET || LOG-END-OFFSET || LAG      || OWNER
test-consumer-group || test || 0 || 10 || 10 || 0 || consumer-1_/10.210.223.170

Upvotes: 10

secfree
secfree

Reputation: 4647

  1. Default, kafka-console-consumer.sh will create a random group.
  2. If you want to specify the group name, you can:
    1. Add group.id=group_name to a local file filename
    2. Use --consumer.config filename option of kafka-console-consumer.sh to set group
  3. You can check your groups at zookeeper's /consumers/ directory.

Refer: kafka/core/src/main/scala/kafka/tools/ConsoleConsumer.scala

Upvotes: 18

Related Questions