CODEWITHSUNDEEP
apache-flink
Tom
Tom

Reputation: 6332

Why should user have to set parallelism explicitly

I kicked off a flink application with n TaskManagers and s slots for each TaskManager, so that, My application will have n*s slots.

That means, flink could be able to run n*s subtasks at most at the same time. But why flink doesn't try to use most resources to run as many subtasks as possible, and bother end users to set the parallelism explicitly?

For the flink beginners that don't know the parallelism setting(default is 1), it will always run only one subtask even given more resources!

I would like to know the design considerations here, thanks!

Upvotes: 0

Views: 48

Answers (1)

Fabian Hueske
Fabian Hueske

Reputation: 18987

A Flink cluster can also be used by multiple users or a single user can run multiple jobs on a cluster. Such clusters are not sized to run a single job but to run multiple jobs. In such environments its not desirable if jobs grab all available resources by default.

Upvotes: 1

Related Questions