CODEWITHSUNDEEP
springspring-bootapache-kafkaapache-kafka-streams
losingsleeep
losingsleeep

Reputation: 1879

Multiple topologies in Spring Kafka streams

What's the easiest or best way to create multiple topologies in a Spring Boot Kafka streams application?

Is it possible to use the same default StreamBuilder bean? Or if I need to create a new StreamBuilder, I don't want to configure it from scratch and / or handle the stream lifecycle myself.

My actual use case is to consume the input topic, and create a Global KTable from that topic too, which is apparently not possible as same topic cannot be registered twice as a source in a Topology.

Upvotes: 1

Views: 61

Answers (1)

Vytautas Šerėnas
Vytautas Šerėnas

Reputation: 327

In order to solve your issue, you don't need multiple topologies. The problem you are facing "apparently not possible as same topic cannot be registered twice", - can be solved by reusing same Stream as a variable:

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> myStream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));

myStream.toTable()... // do global table stuff
myStream.filter(...).map(...).to(ANOTHER_TOPIC);
builder.build();

I can't provide SpringBoot abstraction example, but approach should be similar - reuse same object, pass it via container.

Upvotes: 0

Related Questions