CODEWITHSUNDEEP
apache-flinkflink-streamingflink-cepflink-sql
user3107673
user3107673

Reputation: 443

Flink dynamic scaling 1.5

This is regarding dynamic rescaling in Flink 1.5

I am using Yarn for running Flink jobs. I start these jobs with a static resource. Is there any option to scale out these job by itself in specific conditions like if there's a memory issues.

In Flink 1.5 release notes -

Applications can be rescaled without manually triggering a savepoint. Under the hood, Flink will still take a savepoint, stop the application, and rescale it to the new parallelism.

This means that I will have to monitor my jobs memory and will have to trigger rescale manually. Is these any workaround to handle this.

Upvotes: 2

Views: 634

Answers (2)

Rafi Aroch
Rafi Aroch

Reputation: 416

You would still need to monitor your application, but the rescaling can be done easily by running:

./bin/flink modify -p <NEW-PARALLELISM>

Upvotes: 4

gcandal
gcandal

Reputation: 957

As of 1.5 Flink doesn't support what you want. The process for rescaling a job is:

  1. Start a job with parallelism initialParallelism and maxParallelism maxParallelism.
  2. Manually stop it.
  3. Rerun it with initialParallelism <= parallelism <= maxParallelism.

Upvotes: 3

Related Questions