CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Developer
Developer

Reputation: 259

Routing messages to different Kafka Topic based on payload

My application is getting messages from IBM MQ and based on the contents I have to route it to different Kafka topics.

Kafka MQ source connector can bring events from MQ to 1 kafka topic, can we do message based routing inside the Kafka MQ connector?

Or do we have to write a KStream app to route based on content payload

Upvotes: 0

Views: 3251

Answers (2)

rerorero
rerorero

Reputation: 11

Recently I wrote an SMT for similar case: https://github.com/rerorero/kafka-smt-jsonpath-tools

You can route topics using JsonPath-like expression like this:

{
  ...

  "transforms": "route",
  "transforms.route.type": "com.github.rerorero.kafka.smt.PayloadBasisRouter$Value",
  "transforms.route.replacement": "topic-{$.user.name}"
}

Upvotes: 1

Robin Moffatt
Robin Moffatt

Reputation: 32130

The Single Message Transform API would enable you to change the topic based on the contents of a message, but AFAIK there is not one that exists already so you'd need to write your own.

The alternative is, as you say, to use Kafka Streams or KSQL to do the routing.

This article may also be useful: https://www.confluent.io/blog/putting-events-in-their-place-with-dynamic-routing

Upvotes: 0

Related Questions