CODEWITHSUNDEEP
apache-flinkflink-streaming
user2813148
user2813148

Reputation:

Flink autoscaling and max parallelism

Quote from javadoc on StreamExecutionEnvironment.setMaxParallelism: The maximum degree of parallelism specifies the upper limit for dynamic scaling.

Which exactly dynamic scaling is meant here? I couldn't find any empirical evidence of operator auto scaling: whatever number of free slots you have, and no matter how big is maxParallelism, and how many logical partitions is there, the actual parallelism (according to the web ui) is always the one that was set through a setParallelism

Also, according to this the most accepted and never challenged answer https://stackoverflow.com/a/43493109/2813148 there's no such thing as dynamic scaling in Flink.

So is there any? Or the javadoc is misleading (or what's the meaning of "dynamic" there)? If there's none, are there any plans for this feature?

Upvotes: 5

Views: 2563

Answers (1)

Fabian Hueske
Fabian Hueske

Reputation: 18987

Flink (in version 1.5.0) does not support dynamic scaling yet.

However, job can be manually scaled (or by an external service) by taking a savepoint, stopping the running job, and restarting the job with an adjusted (smaller or larger) parallelism. However, the new parallelism can be at most the previously configured max-parallelism. Once a job was started, the max-parallelism is baked into the savepoints and cannot be changed anymore.

Support for dynamic scaling is on the roadmap. Since version 1.5.0 (released in May 2018), Flink supports dynamic resource allocation from resource managers such as Yarn and Mesos. This is an important step towards dynamic scaling. In fact, an experimental version of this feature has been demonstrated at Flink Forward SF 2018 in April 2018.

Upvotes: 4

Related Questions