CODEWITHSUNDEEP
apache-flink
tashoyan
tashoyan

Reputation: 428

Custom configuration file flink-conf.yaml

I need to specify different Flink settings for different applications. In other words, each application should run with its custom file flink-conf.yaml. What is the proper way to do it?

I found some old recommendations to declare FLINK_CONF_DIR pointing to a custom directory with Flink configuration files (for example: How could I override configuration value in Apache Flink?). However, the official Flink documentation does not mention the FLINK_CONF_DIR variable at all (as of Flink 1.13). Therefore I have doubts, that this way is officially recommended and supported by Flink developers.

UPDATE 1: Details on application running

I am running Flink on YARN in the Application mode. Here is how I launch the application:

"$flink_home/bin/flink" \
  run-application \
  --target yarn-application \
  --class com.example.App1

The out-of-the-box Flink configuration is located in the $flink_home/conf directory. As I have several applications App1, App2, ..., I want them to use their respective Flink configurations instead of the out-of-the-box configuration.

Upvotes: 0

Views: 3853

Answers (1)

David Anderson
David Anderson

Reputation: 43707

TL;DR: The paragraph about FLINK_CONF_DIR was accidentally removed when the Flink on YARN docs were rewritten for the Flink 1.12 release. It is still the intended and supported way to establish per-application settings in YARN clusters.

Other ways to override the configuration:

You can override the settings specified in the cluster's flink-conf.yaml file with settings you specify on the command line, as described in this answer.

You can also override specific settings from the global configuration in your code, e.g.:

Configuration conf = new Configuration();
conf.setString("state.backend", "filesystem");
env = StreamExecutionEnvironment.getExecutionEnvironment(conf);

You can also load all of the settings in a flink-conf.yaml file from your application code, via

FileSystem.initialize(GlobalConfiguration.loadConfiguration("/path/to/conf/directory"));

And with Kubernetes you can mount different ConfigMaps for different applications.

Upvotes: 2

Related Questions