CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Chris W.
Chris W.

Reputation: 2376

Two strange bytes at the beginning of each message of a Kafka message produced by my Kafka Connector

I develop a Kafka connector which simply creates messages for each line in a file retrieved from an external API. It works nicely but now I try to consume the messages and I have two strange bytes at the beginning of each value. I can reproduce the problem with the console consumer and with my kafka stream processor.

�168410002,OpenX Market,459980962,OpenX_Bidder_Order_merkur_bidder_800x250,313115722,OpenX_Bidder_ANY_LI_merkur_800x250_550,106800839362,OpenX_Bidder_Creative_merkur_800x250_2,10

The source files are good and even printlns before creating the SourceRecord don't show these two bytes. I used a struct with one field before and now use a simple String schema but I still have the same problem:

def convert(line: String, ...) = {
...
val record = new SourceRecord(
  Partition.sole(partition),
  offset.forConnectApi,
  topic,
  Schema.STRING_SCHEMA,
  line
)
...

So in the above code, if I add println(line) no strange chars are shown.

Upvotes: 3

Views: 1535

Answers (1)

dawsaw
dawsaw

Reputation: 2313

It looks like you used the AvroConverter or the JsonConverter in your connector. Try using the StringConverter that ships with Kafka in your key.converter and value.converter in the worker for connect. That will encode the data as strings that shouldn't have this extra stuff in it.

Upvotes: 4

Related Questions