CODEWITHSUNDEEP
javaspringkotlinapache-kafkaspring-kafka
pdeva
pdeva

Reputation: 45571

how to compress data in producers when using spring kafka

I am currently sending data using spring-kafka like this:

val json = objectWriter.writeValueAsString(obj)
kafkaTemplate.send(topic, json)

How do i tell KafkaTemplate to use compress the json using snappy before sending?

Upvotes: 7

Views: 10074

Answers (2)

Asif Malek
Asif Malek

Reputation: 307

1.) Compression at broker level.

You can start your zookeeper server with --config compression.type=gzip at the end of the command or you can add this property in zookeeper config file.

2.) Compression at producer level

Set the property compression.type = gzip in config/producer.properties of your producer or start your producer with below property

--compression-codec 'gzip', 'snappy', 'lz4', or 'zstd'.
If specified without value, then it
defaults to 'gzip'

Upvotes: 1

Ryuzaki L
Ryuzaki L

Reputation: 40078

In apache kafka there is producer config compression.type with valid values

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd

So you can set in producer configs

ProducerConfig.COMPRESSION_TYPE_CONFIG "snappy"

or by using properties 

spring.kafka.producer.compression-type= # Compression type for all data generated by the producer.

Upvotes: 9

Related Questions