CODEWITHSUNDEEP
apache-flinkmqtt
YT Q
YT Q

Reputation: 41

How Apache Flink manages MQTT consumer offsets

I'm using MQTT consumer as my flink job's data source. I'm wondering how to save the data offsets into checkpoint to ensure that no data lost when flink cluster restarts after a failure. I've see lots of articles introducing how apache flink manages kafka consumer offsets. Does anyone know whether apache flink has its own function to manage MQTT consumer? Thanks.

Upvotes: 1

Views: 1313

Answers (2)

Niko
Niko

Reputation: 682

You shoul read state-backends part of documentation. And checkpoints section.

When checkpointing is enabled, managed state is persisted to ensure consistent recovery in case of failures. Where the state is persisted during checkpointing depends on the chosen Checkpoint Storage.

Upvotes: 0

Martijn Visser
Martijn Visser

Reputation: 2108

If you have a MQTT consumer, you should make sure it uses the Data Source API. You can read about that on https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ - That also includes how to work integrate with checkpointing. You can also read the details in FLIP-27 https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface

Upvotes: 0

Related Questions