CODEWITHSUNDEEP
apache-kafkaapache-kafka-streamskafka-rest
OldDave2019
OldDave2019

Reputation: 33

How do you set Kafka Rest Proxy Key Schema with JSON values?

I'm using Kafka 1.1 and Kafka Rest Proxy 4.1.2. I have been keying records from inside Kafka streams using String keys. I want to use Rest Proxy to insert records to be joined, but the keys are getting escaped quotation marks put around them.

I'm sending a POST request to /topics/{someTopic} with Content-Type: application/vnd.kafka.json.v2+json which causes the issue.

With Content-Type: application/vnd.kafka.avro.v2+json and key_schema type: string, the keys do not have extra quotation marks around them, but I'd rather send json values.

This is what I am sending to the /topics endpoint.

{
    "records": [
        { 
            "key": "abc", 
            "value": {"animal": "dog"} 
        }
    ]
}

When I stream the data in Kafka streams, the key is coming out as \"abc\", and obviously isn't being joined with records with string keys abc.

Is there a way to specify a key schema with json values, so that my keys don't get escaped quotation marks around them?

Upvotes: 3

Views: 5416

Answers (2)

user3076357
user3076357

Reputation: 64

When using the Content-Type: application/vnd.kafka.json.v2+json header, a JSON key will have escaped quotes around all strings so that it properly deserializes in a streams app. When using a simple key, strings do seem to be escape quoted while numeric keys are unmodified.

The Content-Type: application/vnd.kafka.binary.v2+json will produce your key value pairs exactly as you give them, without adding escaped quotes to string keys. You just need to base64 encode your keys and values.

Your example body becomes:

{
    "records": [{
        "key": "YWJj",
        "value": "eyJhbmltYWwiOiJkb2cifQ=="
    }]
}

Upvotes: 2

Rohit Yadav
Rohit Yadav

Reputation: 2568

As per the Kafka docs, the format of the message is correct for json. I think,you should try with the following message header. 

   Content-Type: application/vnd.kafka.json.v2+json
   Accept: application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json

I would suggest, you to go through the following Kafka document.

https://docs.confluent.io/current/kafka-rest/api.html

Upvotes: 2

Related Questions