CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
Lucian
Lucian

Reputation: 824

Multiple Kafka Streams with different topics ApplicationId

Given a single component which can have multiple instances, and the following structure:

  1. Stream1[Topic1, destination1]
  2. Stream2[Topic2, destination2]

where destination is a Queue and all the links will be 1:1.

Do we need to set the same applicationId for each KafkaStream? It is known that applicationId will generate client.id and group.id which are important for how the partitions are assigned. Couldn't find anything in the official documentation.

Upvotes: 1

Views: 796

Answers (2)

Matthias J. Sax
Matthias J. Sax

Reputation: 62285

If your program is different (ie, different Topology), you need to use different application.id config. Using the same application.id config requires that all instances execute the exact same Topology.

Of course, you could also build a single Topology that processed both topics at once:

StreamsBuilder builder = ...

builder.stream("topic1")...to("destination1");
builder.stream("topic2")...to("destination2");

Upvotes: 1

OneCricketeer
OneCricketeer

Reputation: 191671

You can run two applications in the same JVM process with separate threads for starting both topologies, or you can simply run two independent JVM processes. Both cases, use different ids.

Or you can run one process (one id), subscribe to both topics, but use branch operator to separate streams by topic names.

Upvotes: 1

Related Questions