CODEWITHSUNDEEP
javaapache-kafkaapache-kafka-streams
Luca Tampellini
Luca Tampellini

Reputation: 1849

Kafka Streams - one on two does not operate - wrong partition assignments

In an application of my company, in order to apply a few transformations to 2 group of messages called LIVE and PRE-MATCH, we create 2 Kafka Streams, one for each of these groups. Both of these streams are started correctly, however one of them which is always the second created on a time-based order, does not operate any change: by this I mean that it doesn't consume anything from the start topic.

An analysis of the logs, shows that there must be some issue with the configuration of topics and partitions:

2018-09-26 11:36:59,963 [INFO ] [] [StreamThread-2] [o.a.k.c.c.i.ConsumerCoordinator] - Setting newly assigned partitions [] for group Union
2018-09-26 11:36:59,963 [INFO ] [] [StreamThread-1] [o.a.k.c.c.i.ConsumerCoordinator] - Setting newly assigned partitions [betgenius.integralfeed.live-0] for group Union

As you can see in the portion of log above, the StreamThread-1, associated to LIVE feeds, gets correct assignment of partitions mapped to the correct source-topic; StreamThread-2, which is associated in this case to PRE-MATCH feeds, seems to receive a partition assignment to an empty source-topic and is actually the one that does not operate.

If i artificially force the PRE-MATCH stream to be created before the LIVE one, the situation is reversed.

The code used to create the 2 streams is absolutely identical, I just change the start topic and the end topics.

Is it possible that there are not enough resources available on my configuration of Kafka?

Upvotes: 1

Views: 518

Answers (1)

Vasyl Sarzhynskyi
Vasyl Sarzhynskyi

Reputation: 3955

according to provided logs, seems you use the same groupId Union for both Kafka streams, but they should be unique for each stream. so make sure you provided different values for settings property application.id.

application.id - identifier for the stream processing application, must be unique within the Kafka cluster.

Upvotes: 2

Related Questions