CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
user1028741
user1028741

Reputation: 2815

Kafka Streams - floods kafka logs

I'm using Kafka Streams, and I've noticed that it makes my kafka logs a lot of logs messages such as:

[2019-04-17 09:06:16,541] INFO [Log partition=my-application-KSTREAM-AGGREGATE-STATE-STORE-0000000076-repartition-0, dir=/opt/kafka/data/logs] Incrementing log start offset to 316423 (kafka.log.Log) [2019-04-17 09:06:16,545] INFO [Log partition=my-application-KSTREAM-AGGREGATE-STATE-STORE-0000000033-repartition-2, dir=/opt/kafka/data/logs] Incrementing log start offset to 3394 (kafka.log.Log) 
[2019-04-17 09:06:16,546] INFO [Log partition=my-application-KSTREAM-AGGREGATE-STATE-STORE-0000000012-repartition-1, dir=/opt/kafka/data/logs] Incrementing log start offset to 3102 (kafka.log.Log) 
[2019-04-17 09:06:16,549] INFO [Log partition=my-application-KSTREAM-AGGREGATE-STATE-STORE-0000000067-repartition-3, dir=/opt/kafka/data/logs] Incrementing log start offset to 4168 (kafka.log.Log) 
[2019-04-17 09:06:16,551] INFO [Log partition=my-application-KSTREAM-AGGREGATE-STATE-STORE-0000000086-repartition-3, dir=/opt/kafka/data/logs] Incrementing log start offset to 329670 (kafka.log.Log)

This is really disturbing because I it floods the logs so I cannot see anything else (also consuming resources).

Why does it happen for (some of) the Kafka Streams internal topics, but not for other topics?

How can I disable it?

Upvotes: 3

Views: 1275

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

Kafka Streams purges repartition topics aggressively after the data is processed to free up storage space in the Kafka cluster (cf. https://issues.apache.org/jira/browse/KAFKA-6150). It's not possible to disable this.

As a workaround, you could reconfigure the repartition topic configuration parameters segment.bytes and segment.ms and set them to higher values. Note, that this will make data purging less effective, though, resulting in more disk usage in the Kafka cluster.

Upvotes: 1

Related Questions