CODEWITHSUNDEEP
apache-kafkaavroapache-nifi
jmarco10
jmarco10

Reputation: 525

Read Avro Buffer encoded messages with ConsumeKafka (NIFI)

I am new to NIFI(and not much experience with Kafka), and I am trying to consume the messages that the producer is generating. To do this job, I am using the the ConsumeKafka processor on NIFI.

The messages are arriving (I can see them on the queue), but when I check the queue, and try to view the messages, I can only see the content with hex format (f.e: in original format I can see a message that says: No viewer is registered for this content type).

The messages that the producer is sending are encoded avro buffer (this is the reference I have taken: https://blog.mimacom.com/apache-kafka-with-node-js/) And when I check the consumer from the console, each message has this format:

02018-09-21T08:37:44.587Z      @02018-09-21T08:37:44.587Z      @

I have read that the processor UpdateRecord can help to change the hex code to plain text, but I can't make it happen.

How can I configure this UpdateRecord processor?

Chears

Upvotes: 0

Views: 1313

Answers (1)

Sivaprasanna Sethuraman
Sivaprasanna Sethuraman

Reputation: 4132

  • Instead of ConsumeKafka, it is better to use ConsumeKafkaRecord processor appropriate to the Kafka version you're using and configure Record Reader with an AvroReader and set Record Writer to the writer of your choice.
  • Once that's done, you have to configure the AvroReader controller service with a Schema registry. You can use AvroSchemaRegistry where you would specify the schema for the Avro messages that you're receiving in Kafka.

A quick look at this tutorial would help you achieve what you want: https://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries

Upvotes: 4

Related Questions