CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Wrapper
Wrapper

Reputation: 932

Create Kafka consumers within a consumer group in such a way to read from two (or more) topics?

Step 1.

I created two topics with different number of partitions, like so:

kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --create --partitions 5 --replication-factor 1

and

kafka-topics.sh --bootstrap-server localhost:9092 --topic second_topic --create --partitions 4 --replication-factor 1

Step 2. - attempt 1.

Create three consumers to listen on two previously created topic.

Execute this command in three different terminals:

kafka-console-consumer.sh --bootstrap-server localhost:9092  --whitelist 'first-topic|second_topic' --property print.offset=true --property print.partition=true --group my-first-application

Describe consumer group:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-first-application

GROUP                TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                           HOST            CLIENT-ID
my-first-application second_topic    0          0               0               0               console-consumer-3db678b0-931c-4e22-9632-040df2d64d62 /127.0.0.1      console-consumer
my-first-application second_topic    1          0               0               0               console-consumer-3db678b0-931c-4e22-9632-040df2d64d62 /127.0.0.1      console-consumer
my-first-application second_topic    3          0               0               0               console-consumer-ae8a6f87-9ac7-4eb5-a9eb-0445254e2d24 /127.0.0.1      console-consumer
my-first-application second_topic    2          0               0               0               console-consumer-58b4d231-58cf-4c63-a3f1-fbe4d50873e3 /127.0.0.1      console-consumer
my-first-application first_topic     2          0               0               0               -                                                     -               -
my-first-application first_topic     1          0               0               0               -                                                     -               -
my-first-application first_topic     4          0               0               0               -                                                     -               -
my-first-application first_topic     0          0               0               0               -                                                     -               -
my-first-application first_topic     3          0               0               0               -                                                     -               -

Only partitions in second_topic got distributed for given consumer group.

Step 2. - attempt 2.

Execute this command in three different terminals, after killing active consumers:

kafka-console-consumer.sh --bootstrap-server localhost:9092  --whitelist 'first-topic,second_topic' --property print.offset=true --property print.partition=true --group my-first-application

Describe group:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-first-application

GROUP                TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
my-first-application second_topic    0          0               0               0               -               -               -
my-first-application first_topic     1          0               0               0               -               -               -
my-first-application second_topic    2          0               0               0               -               -               -
my-first-application first_topic     4          0               0               0               -               -               -
my-first-application first_topic     0          0               0               0               -               -               -
my-first-application second_topic    1          0               0               0               -               -               -
my-first-application first_topic     3          0               0               0               -               -               -
my-first-application second_topic    3          0               0               0               -               -               -
my-first-application first_topic     2          0               0               0               -               -               -

Upvotes: 0

Views: 282

Answers (1)

Ankit Sahay
Ankit Sahay

Reputation: 2043

It's because of a typo. You have created the topic name as "first_topic" But during whitelisting you have used "first-topic"

Upvotes: 2

Related Questions