CODEWITHSUNDEEP
apache-kafkaavroapache-kafka-streamsapache-kafka-connectconfluent-platform
TechEnthusiast
TechEnthusiast

Reputation: 1895

Confluent - Splitting Avro messages from one kafka topic into multiple kafka topics

We have an incoming kafka topic with multiple Avro schema based messages serialized into it.

We need to split the messages in Avro format into multiple other kafka topics based on certain value of a common schema attribute.

                             |------> [OUTGOING TOPIC(AVRO) - A] 
[INCOMING TOPIC(AVRO)] ----->|------> [OUTGOING TOPIC(AVRO) - B]
                             |------> [OUTGOING TOPIC(AVRO) - C]

Would like to understand how to achieve it while avoid building an intermediate client to do this splitting/routing in confluent platform.

I explored the kafka connector but did not find an existing connector which does this work.

Upvotes: 1

Views: 477

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

You can write a Kafka Streams application and use branch():

KStream input = builder.stream("topic");
KStream[] splitStreams = input.branch(...);
splitStream[0].to("output-topic-1");
splitStream[1].to("output-topic-2");
// etc.

Upvotes: 1

Related Questions