CODEWITHSUNDEEP
apache-flinkflink-sql
Norman He
Norman He

Reputation: 31

How to configure flink SQL client to submit job with checkpointing enabled?

What is an example of a flinksql-client config file with checkpointing enabled?

When I submit streaming job through flink sql client, the checkpoint is not enabled.

Upvotes: 2

Views: 2808

Answers (3)

tom yang
tom yang

Reputation: 39

you can actually pass in an init sql file when calling sql client without having to update the cluster conf.yml

./sql-client

 -i,--init <initialization file>            Script file that used to init
                                            the session context. If get
                                            error in execution, the sql
                                            client will exit. Notice it's
                                            not allowed to add query or
                                            insert into the init file.

in your file you can then specify

SET 'execution.checkpointing.interval' = '1min';

https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/

Upvotes: 0

Alex
Alex

Reputation: 21

As far as I see in the docs you should be able to do it from the sql client by setting the properties like this:

SET 'state.checkpoints.dir' = 'hdfs:///bar/foo/';
SET 'execution.checkpointing.mode' = 'EXACTLY_ONCE';
SET 'execution.checkpointing.interval' = '30min';
SET 'execution.checkpointing.min-pause' = '20min';
SET 'execution.checkpointing.max-concurrent-checkpoints' = '1';
SET 'execution.checkpointing.prefer-checkpoint-for-recovery' = 'true';

Upvotes: 2

David Anderson
David Anderson

Reputation: 43454

The checkpointing configuration can not be set in flink sql client config file, but it can be set in the cluster configuration file (flink-conf.yaml).

At a minimum you should configure execution.checkpointing.interval, state.backend, and state.checkpoints.dir.

Something like this, for example:

execution.checkpointing.interval: 10000
state.backend: filesystem
state.checkpoints.dir: file:///tmp/flink-checkpoints-directory
state.savepoints.dir: file:///tmp/flink-savepoints-directory

See the configuration docs for checkpointing, fault tolerance, and state backends for more details.

Upvotes: 3

Related Questions