CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-apiapache-kafka-streams
Dede
Dede

Reputation: 1640

Kafka Streams: how to convert data types?

I am using Kafka's Streams API with topology builder. I would like to know how I can do to have a processor that can convert one data type to another, so the next processor in the pipeline can use it.

As a simple use case :

[topic]--(string)-->[processor: parse json]--(object)-->[processor 2]--(object)-->[sink]

Any idea ?

Upvotes: 2

Views: 2702

Answers (1)

miguno
miguno

Reputation: 15057

I assume you want to convert the message values in a Kafka topic from String to JSON.

You only need two parts:

  • The code to convert String to JSON (or Pojo). Pick whatever you need here, e.g. there are a couple of Java libraries available to make this easy.
  • In Kafka Streams, define (1) a value serde for String for reading from your source topic, and (2) define a corresponding value serde for writing the JSON data (or Pojo) to the destination topic. Serdes are required to materialize your data when/where needed (e.g., writing your Pojos to Kafka requires materialization).

See the example code under https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview for how to e.g. use JSON with Apache Kafka's Streams API.

Upvotes: 2

Related Questions