CODEWITHSUNDEEP
apache-kafka
Girish Chafle
Girish Chafle

Reputation: 123

How can I verify if compression is working correctly in Kafka 0.8.2.2?

I am using Kafka 0.8.2.2 and am trying to set up compression. I am providing the compression-codec (gzip) as an argument to the console producer like below.

./kafka-console-producer.sh --broker-list localhost:171 --compression-codec gzip --topic testTopic

Questions Is this the only place where I need to specify compression? How do I verify if compression is indeed taking place? How do I quantify the benefit I am getting from compression? What files (.index, .log) I should look for and compare the sizes with and without compression to estimate the benefit?

Upvotes: 11

Views: 10879

Answers (2)

Adomas
Adomas

Reputation: 506

In case you need to verify if compression is working where you do not have access to kafka CMD tools (like AWS MKS) Kowl kafka UI client (https://github.com/theurichde/kowl) can be used. Run it against your remote kafka broker, open the topic that you need, double click on the message and you will see listed compression.

For cases where you have access to kafka cmd tools (or locally) - answer by Marina is valid.

Upvotes: 0

Marina
Marina

Reputation: 4064

How to verify if compression is happening?

Use DumpLogSegments tool and substitute your dir location / log file name (default log.dir is /tmp/kafka-logs)

bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /your_kafka_logs_dir/your_topic-your_partition/00000000000000000000.log --print-data-log | grep compresscodec

You will see something like below:

baseOffset: 0 lastOffset: 0 count: 1 ... compresscodec: NONE ...
baseOffset: 1 lastOffset: 1 count: 1 ... compresscodec: GZIP ...
baseOffset: 2 lastOffset: 2 count: 1 ... compresscodec: SNAPPY ...
baseOffset: 3 lastOffset: 3 count: 1 ... compresscodec: LZ4 ...

More info can be found in documentation here https://kafka.apache.org/documentation/#design_compression

Upvotes: 27

Related Questions