CODEWITHSUNDEEP
apache-kafka
Anirban
Anirban

Reputation: 277

How to handle compressed data using Kafka console producer

I have a file with JSON data. How can I send the data to a topic using console producer in compressed format (gzip) ? Please note - I can do it by writing a custom producer in Java. My question is specifically about console producer.

Upvotes: 1

Views: 1055

Answers (1)

andybryant
andybryant

Reputation: 331

If you type kafka-console-producer on it's own you get the full list of options. You'll see it includes the following...

--compression-codec [String:             The compression codec: either 'none',  
  compression-codec]                       'gzip', 'snappy', 'lz4', or 'zstd'.  
                                           If specified without value, then it  
                                           defaults to 'gzip'

so you could run something like...

cat my-data.json | kafka-console-producer --broker-list kafka:9092 --topic json-topic --compression-codec gzip

Upvotes: 4

Related Questions