Reputation: 195
I have a flink pipeline running with some filters,maps,aggregator and windows operators. RocksDB backend with incremental checkpointing is enabled.
My checkpoint data size is growing gradually. I understand that once compaction starts check point size will reduce gradually. I want to test and prove this.
For that I configured the following values in my flink conf yaml.
rocksdb.compaction.level.max-size-level-base : 4096
rocksdb.compaction.level.target-file-size-base : 2048
rocksdb.writebuffer.size : 2048
Though I see in task manager logs that these properties are recognized. Check point size is not reducing even after 4 MB. My initial check point size was ~400 KB.
Anything I am missing here? Do I need to enable any other properties?
I am not able to find if the rocksDB compaction happened or not. I want to enable rocks DB native metrics, so that I can monitor them in either flink web UI or a REST API? How to do that?
(I don't have any RichFunctions in my code to configure the metrics programmatically as told here https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/.)
(I understand that if check point size is not reducing there are some active state and we need to configure StateTTL. Before adding stateTTL i want to prove that compaction is not enough in my case)
Any help would really be appreciated.
Upvotes: 0
Views: 299
Reputation: 43454
For each RocksDB metric you wish to enable, set the corresponding config variable to true in your config. E.g.,
state.backend.rocksdb.metrics.estimate-live-data-size: true
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
For the config settings you mention, you need to include the complete name of each setting, and you should include units. E.g.,
state.backend.rocksdb.compaction.level.max-size-level-base: 4mb
Just because the settings are echoed into the task manager logs doesn't mean they were recognized and applied.
Upvotes: 1