CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectsmt
Ckey
Ckey

Reputation: 21

How get payloads from jdbc source connector in the kafka topic as a nested json object?

Kafka topic is showing payloads as such :

{
  "id": 73,
  "message": "{\"id\":3,\"firstName\":\"firstname\",\"lastName\":\"lastname\",\"emailAddress\":\"[email protected]\"}"
}

How do I get the payload to become like this (I mean not converting the inner fields like message to string values):

{
  "id": 73,
  "message": {"id":3,"firstName":"firstname","lastName":"lastname","emailAddress":"[email protected]"}
}

Is it using transformers or using converters ?

What I tried: I tried changing converters types (json, avro) with no luck. I tried creating my own SMT, since there's no such one on github or provided by apache.kafka , with no luck also.

Upvotes: 0

Views: 375

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191914

A converter is only responsible for deserializing Kafka records, not also parsing them.

There's no built in JSON parser SMT because this would require reflection to parse the JSON and create a Struct/Schema object in the Connect API. The Struct is easy, but the schema isn't without more information such as nullability of fields, if some strings are actually enums, or guarantee that all "message" payloads have a consistent form (since it's only able to access one record at a time), etc.

For example - https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-json-schema/transformations/FromJson.html

tried Avro

The format doesn't matter since the JDBC source docs have an explicit table about column data type mappings. If you have TEXT/VARCHAR columns, you're going to get strings in Kafka.

tried creating my own SMT

You'll need to show that code if you want help with it.

Otherwise, just accept the data and use json parsers in your consumer code, or use ksqlDB / kafka streams to create another, parsed topic

Upvotes: 0

Related Questions