CODEWITHSUNDEEP
apache-flinkdistributed-computing
Robert Lugg
Robert Lugg

Reputation: 1192

Set taskmanager.numberOfTaskSlots on a per-worker in Flink

I am working in a standalone cluster. All servers can run taskmanager.sh. My cluster is heterogeneous with different core counts and memory. The code that I run (Apache Beam ParDo's) is a python single-threaded operation.

I would like to set the number of task slots to be different for each server. How can I have flink run with different slots for different servers?

Should I instead start multiple taskmanager.sh 's per server?

Ref: https://stackoverflow.com/a/54117789/2184122

I also hope that this statement:

A Flink cluster needs exactly as many task slots as the highest parallelism
used in the job. No need to calculate how many tasks (with varying parallelism)
a program contains in total.

means total number of slots (ie roughly #Hosts * #Cores/host). Is my understanding correct??

Upvotes: 1

Views: 1926

Answers (1)

David Anderson
David Anderson

Reputation: 43499

It is generally recommended to run with one slot per taskmanager as a starting point (particularly in containerized environments, though that doesn't sound like your case). This provides better isolation, and keeps the heap sizes smaller, which minimizes GC impact. At some point, though, as you scale up, it becomes unworkable to have the job manager coordinating so many TMs, and you're better off running with multiple slots per TM.

Also, the scheduler only thinks in terms of slots, not TMs, so you want to keep the slots uniform in terms of their resources. A typical starting point is to allocate as many slots per machine as there are cores, though for compute intensive workloads you might want more cores per slot. And for workloads that require very little CPU, it can make sense to go in the other direction, and allocate more slots than cores.

Upvotes: 2

Related Questions