CODEWITHSUNDEEP
dockerdocker-composeyamlapache-flink
Dharam
Dharam

Reputation: 267

How to pass Configuration parameters to Flink Docker image via docker-compose.yml?

I want to pass below configuration parameters to docker-compose.yml for flink docker image.

state.checkpoints.dir: s3://.../flink/checkpoint
state.savepoints.dir: s3://.../flink/savepoint
s3.access-key: ...
s3.secret-key: ...
jobmanager.execution.failover-strategy: region

Can someone help me on how to do it ?

docker image - flink:1.9.0

Upvotes: 4

Views: 4042

Answers (2)

TomTom
TomTom

Reputation: 41

In case anyone is looking for the answer, add double quote around the whole entry of FLINK_PROPERTIES to escape the colon:

jobmanager:
    image: flink:1.9-scala_2.11
    command: "jobmanager.sh start-foreground"
    ports:
      - 8081:8081
    volumes:
      - ./conf:/opt/flink/conf
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
      - "FLINK_PROPERTIES=a: b \nc: d"

Upvotes: 3

Fabian Hueske
Fabian Hueske

Reputation: 18987

Mount a config folder into the container

You can create a volume that mounts a folder from the host system to the location of the Flink configuration in the Docker container.

The Flink Operations Playground is doing this to pass custom configuration to the Flink Docker image.

The entry for the JobManager in the docker-compose.yaml could look like this:

  jobmanager:
    image: flink:1.9-scala_2.11
    command: "jobmanager.sh start-foreground"
    ports:
      - 8081:8081
    volumes:
      - ./conf:/opt/flink/conf
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

It would mount the ./conf folder (which should be next to the docker-compose.yaml file) to /opt/flink/conf in the Docker container which is the location that FLink takes its configuration from.

Pass arguments via an environment variable

An alternative could be to pass configuration arguments via an environment variable.

The docker-entrypoint.sh script, which is used to start the Flink container, seems to append the content of the FLINK_PROPERTIES variable to the configuration file. However, I haven't used that approach yet.

Upvotes: 2

Related Questions