CODEWITHSUNDEEP
apache-kafka-streamssliding-window
JInu Thomas
JInu Thomas

Reputation: 61

Kafka Streams with large sliding windows

I need to show usage stats at any points on time for last 3 months, 6 months and 1 year. I am planning to use the KStream sliding windows for the durations mentioned above. Most of the examples I see are using durations in minutes or seconds. I would like to know is that OK to use the bigger time duration for sliding windows? Any performance impact? Any specific configuration It should use to get optimum performance?

Thanks, Jinu

Upvotes: 1

Views: 991

Answers (1)

wcarlson
wcarlson

Reputation: 226

It will really depend on the density of the data and what kind of aggregations you are doing. It could end up with very large number of windows updating and not closing since the end time is so far out. Also if it is too heavy I am not sure the state stores could handle it. But with the correct load and retention times I don't see an obvious reason it wouldn't work.

Edit: If you do end up trying it I would be very interested in seeing how it works out.

Upvotes: 1

Related Questions