Confused about FLINK task slot

Question

I know a task manager could have several task slots.

But, what is a task slot ? A JVM process or an object in memory or a thread?

Answer 1

In a nutshell - a task slot is a resource management unit, where operator chains are executed in dedicated threads with state per the subtask. A slot obtains a portion of memory for the subtask execution and uses it mainly to keep states. A taskSkot has its own thread pool, which gets a chain of operators for execution. The chain is executed in a thread with exclusive access to its state. Usually, all operators of a chain of a graph (from source to sink) are executed in a thread and using resources of a task slot in the same TM but it is not obligatory.

Answer 2

I would like to address some inconsistencies in the previous answers:

TaskSlot is a Thread

With default settings TaskSlot != Thread. TaskSlot = Thread only if slot sharing is disabled. Slot sharing is an optimization that is on by default and, in most cases, you would want to keep it that way. It is more precise to say that each Operator Chain gets a separate Thread [1] spawn for its execution.

a TS represents a fixed subset of resources of the TM

This is not the case. A Task Slot is purely a scheduling concept that gives some control over how many Operators or Operator Chains will be placed on which task manager. There is no guaranteed allocation of a fixed set of resources to a Task Slot. For instance, if you have a Task Manager with 5 cores and 5GB of RAM which is assigned two Task Slots, those slots will be "competing" for both the CPU and the memory - they are not split in any fixed manner.

Answer 3

@Janukowitsch's answer is good. Additionally, a TS represents a fixed subset of resources of the TM. One TS can take more than 1 thread. Actually one subtask/task is executed by one thread. Multiple subtasks/tasks can be deployed in the same slot.

Answer 4

Each worker (TaskManager) is a JVM process, and may execute one or more subtasks in separate threads. To control how many tasks a worker accepts, a worker has so called task slots (at least one).

https://ci.apache.org/projects/flink/flink-docs-master/concepts/runtime.html#task-slots-and-resources

Answer 5

The answer might come late. But:

A Taskmanager (TM) is a JVM process, whereas a Taskslot (TS) is a Thread within the respective JVM process (TM). The managed memory of a TM is equally split up between the TS within a TM. No CPU isolation happens between the slots, just the managed memory is divided. Moreover, TS in the same TM share TCP connections (via multiplexing) and heartbeat messages. They may also share data sets and data structures, thus reducing the per-task overhead.

Source: https://ci.apache.org/projects/flink/flink-docs-release-1.5/concepts/runtime.html#task-slots-and-resources

Answer 6

The number of slots is typically proportional to the number of available CPU cores of each TaskManager. As a general recommendation, the number of available CPU cores is a good default for taskmanager.numberOfTaskSlots.