CODEWITHSUNDEEP
apache-kafka-streams
Holm
Holm

Reputation: 3355

Can I compress intermediate topic (state store) used by KafkaStream in Kafka

When using stateful processor, the states can also be stored in kafka.

For example, those topics with the names as the following: -repartition-x and -changelog-x

Can I compress these topics, for example, with gzip, and KafkaStream will still be able to use them?

Upvotes: 3

Views: 989

Answers (1)

Tim Van Laer
Tim Van Laer

Reputation: 2534

As Mathias pointed out, you can compress messages by setting the compression.type topic config in internal topic configuration. See https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/state/StoreBuilder.html#withLoggingEnabled-java.util.Map-

Alternatively you can also implement a custom Serde which does the compression.

Prefer the former as it will only be compressed in the Kafka topics and not in the state store. This avoids unnecessary compress/uncompress during processing.

Upvotes: 3

Related Questions