CODEWITHSUNDEEP
apache-kafkakafka-producer-api
JL88
JL88

Reputation: 23

What's the difference between setting `compression.type` in Kafka producer vs. defining it in the broker?

I am trying to set compression.type, and currently setting it in my broker configuration. If I do not define the property in my producer, will it take effect? I do not specify any compression.type value or even the property in my producer deployment.

Upvotes: 2

Views: 926

Answers (2)

mazaneicha
mazaneicha

Reputation: 9427

These two properties serve different purposes, see Kafka reference doc.

When defined on a producer side, compression.type codec is used to compress every batch for transmission, and thus to increase channel throughput.

At the topic (broker) level, compression.type defines the codec used to store data in Kafka log, i.e. minimize disk usage. Special value producer allows Kafka to retain original codec set by producer.

Upvotes: 3

maxgruber19
maxgruber19

Reputation: 163

it depends on what you wanna compress. You can compress the data sent and you can compress the data stored in topics. So define it on broker side to configure how data is saved or define it on producer side to configure how data is sent

broker side:

Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd')

refer to confluent documentation

producer side:

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd.

refer to confluent documentation

Upvotes: 0

Related Questions