CODEWITHSUNDEEP
apache-kafka-streams
posthumecaver
posthumecaver

Reputation: 1843

What is a reasonable value for StreamsConfig.COMMIT_INTERVAL_MS_CONFIG for Kafka Streams

I was looking to some confluent examples for the Kafka Streams the different values for configuration value 'StreamsConfig.COMMIT_INTERVAL_MS_CONFIG' confused me little bit.

For ex, in micro service example, 

config.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1); //commit as fast as possible

https://github.com/confluentinc/kafka-streams-examples/blob/5.1.0-post/src/main/java/io/confluent/examples/streams/microservices/util/MicroserviceUtils.java

Another one,

// Records should be flushed every 10 seconds. This is less than the 
default
// in order to keep this example interactive.
streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10 * 
1000);

https://github.com/confluentinc/kafka-streams-examples/blob/5.1.0-post/src/main/java/io/confluent/examples/streams/WordCountLambdaExample.java

Another one,

// Set the commit interval to 500ms so that any changes are flushed 
frequently and the top five
// charts are updated with low latency.
streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 
 500);

https://github.com/confluentinc/kafka-streams-examples/blob/5.1.0-post/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java

In the examples intervals changes from 1ms to 10000ms, what I am really interested is the 1ms in a system that heavy load all the time, could it be dangerous to go 1ms Commit Interval?

Thx for answers..

Upvotes: 1

Views: 1778

Answers (1)

Nishu Tayal
Nishu Tayal

Reputation: 20830

Well, it depends how frequently you want to commit your records. It actually refers to the Record Caching in memory:

https://kafka.apache.org/21/documentation/streams/developer-guide/memory-mgmt.html#record-caches-in-the-dsl

If you want to see each record as output, you can set it to the lowest number. In some scenarios, you may want to get the output for each event, there having lowest number makes sense. But in some scenario, where it is okay to consolidate the events and produce fewer output, you can set it to higher number.

Also be aware, that Record caching is affected by these two configuration:

commit.interval.ms and cache.max.byte.buffering

The semantics of caching is that data is flushed to the state store and forwarded to the next downstream processor node whenever the earliest of commit.interval.ms or cache.max.bytes.buffering (cache pressure) hits.

Upvotes: 3

Related Questions