Banupriya
Banupriya

Reputation: 195

How to test rocksDB compactions in apache flink checkpoints

I have a flink pipeline running with some filters,maps,aggregator and windows operators. RocksDB backend with incremental checkpointing is enabled.

My checkpoint data size is growing gradually. I understand that once compaction starts check point size will reduce gradually. I want to test and prove this.

For that I configured the following values in my flink conf yaml.

rocksdb.compaction.level.max-size-level-base : 4096
rocksdb.compaction.level.target-file-size-base  : 2048
rocksdb.writebuffer.size : 2048

Though I see in task manager logs that these properties are recognized. Check point size is not reducing even after 4 MB. My initial check point size was ~400 KB.

Anything I am missing here? Do I need to enable any other properties?

I am not able to find if the rocksDB compaction happened or not. I want to enable rocks DB native metrics, so that I can monitor them in either flink web UI or a REST API? How to do that?

(I don't have any RichFunctions in my code to configure the metrics programmatically as told here https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/.)

(I understand that if check point size is not reducing there are some active state and we need to configure StateTTL. Before adding stateTTL i want to prove that compaction is not enough in my case)

Any help would really be appreciated.

Upvotes: 0

Views: 299

Answers (1)

David Anderson
David Anderson

Reputation: 43454

For each RocksDB metric you wish to enable, set the corresponding config variable to true in your config. E.g.,

state.backend.rocksdb.metrics.estimate-live-data-size: true

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics

For the config settings you mention, you need to include the complete name of each setting, and you should include units. E.g.,

state.backend.rocksdb.compaction.level.max-size-level-base: 4mb

Just because the settings are echoed into the task manager logs doesn't mean they were recognized and applied.

Upvotes: 1

Related Questions