CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
NoName
NoName

Reputation: 1759

Kafka Streams: Reading from topic with multiple partitions

I have a source topic with 2 partitions and I'm starting 2 kafka streams application with same application.id but different sink topics.

1) Will the 2 application instances receive data from different partitions?

2) If one of the applications is killed, will the other instance automatically consume from both the instances?

3) How can I prove the above scenario?

Upvotes: 0

Views: 1235

Answers (2)

miguno
miguno

Reputation: 15077

1) Will the 2 application instances receive data from different partitions?

Yes.

2) If one of the applications is killed, will the other instance automatically consume from both the instances?

Yes. (But keep in mind that the other instance may now send the output data to its own, rather than the original instance's output topics.)

3) How can I prove the above scenario?

You can inspect the logs of your application (Kafka Streams will log information about topics/partitions/stream tasks for each instances), you can run integration tests, etc.

Upvotes: 0

Alex Ott
Alex Ott

Reputation: 87259

Kafka Streams is using the same consumer library, so behavior should match to what you described - if you have 2 applications, then each will consume from partitions assigned to it, and if one is killed, then the alive will process data from all partitions...

Prove is possible by running applications, for example - in simplest case, just print different prefixes for consumed data, and submit keyed values so you can distinguish when one message is going to one partition from what is sent to another...

Upvotes: 1

Related Questions