Are there limitations in using a State in Apache Flink?

Question

Apache Flink allows me to use a State in a RichMapFunction. I am planning to build a continuously running job which analyses a stream of web events. Part of the processing will be the creation of a session context with session scoped metrics (like nth of the session, duration etc) and additionally a user context.

A session context will timeout after 30 minutes, but a user context may exist for a year to handle returning users.

There will be millions of sessions and users so I would end up in millions of individual states. Every state is just a few KB in size.

• Is this something that can be handled properly with the Flink states?
• How is Flink actually cleaning up deprecated states?
• Would it make sense to think about providing a custom backend to store the state in a KV cluster?

Answer 1

For large state I would recommend using Flink's RocksDBStateBackend. This state backend uses RocksDB to store state. Since RocksDB gracefully spills to disk, it is only limited by your available disk space. Thus, Flink should be able to handle your use case.

At the moment you need to register timers to clean up state. However, with the next Flink release, the community will add clean up for state with TTL. This will then automatically clean up your state when it is expired.

Keeping your state close to your computation with periodic checkpoints which are persisted will keep your application fast. If every state access went to a remote KV cluster, it would considerably slow down the processing.