CODEWITHSUNDEEP
apache-flink
Tom
Tom

Reputation: 6342

How to specify the number of cores for task manager/task slot in flink

Give the following yarn-session command:

yarn-session.sh -d -n 12 -tm 4096 -qu QueueX -s 4 -jm 2048

We specify the memory for the task manager as 4096, and also we specify the number of slots for each task manager as 4. and throughly 1/4 of task manager's memory is allocated to each task slot.

Then, what about the CPU cores?

Can we specify the number of cores for each task manager?(By default, each yarn container will only has one core). And further, can we specify the number of cores for each task slot?

If we don't specify the number of cores, how many cores each task manager / task slot will use.

Upvotes: 0

Views: 2165

Answers (1)

Mikalai Lushchytski
Mikalai Lushchytski

Reputation: 1641

By default, the number of vcores is set to the number of slots per TaskManager, if set, or to 1, otherwise. The number of slots per TaskManager is specified by taskmanager.numberOfTaskSlots parameter in flink/conf/flink-conf.yaml. If you want to allocate a number of vcores for each TaskManager, different from slots number, you can additionally provide the yarn.containers.vcores setting in flink/conf/flink-conf.yaml - the number of virtual cores (vcores) per YARN container.

Upvotes: 1

Related Questions