CODEWITHSUNDEEP
apache-flinkflink-streaming
user2456976
user2456976

Reputation: 1308

Ideal Number of Task Slots

We have developed a Flink application on v1.13.0 and deployed it on Kubernetes that runs a Task Manager instance on a Kubernetes pod. I am not sure how to determine the ideal number of task slots on each Task Manager instance. Should we configure/choose one task slot on each task manager/pod or two slots per Task Manager/pod or more. We currently configured two task slots per Task Manager instance and wondering if that is the right choice/setting. What are the pros and cons of running one task slot vs running two or more slots on a Task Manager/pod.

Upvotes: 1

Views: 1470

Answers (2)

David Anderson
David Anderson

Reputation: 43409

As a general rule, for containerized deployments (like yours), one slot per TM is a good default starting point. This tends to keep the configuration as straightforward as possible.

Upvotes: 2

BenoitParis
BenoitParis

Reputation: 3184

Depends on your expected workload, input, state size.

Is it a batch or a stream?

Batch: is the worload fast enough?

Stream: is the worload backpressuring?

For these throughput limitations, you might want to increase the number of TMs

State size: how are you processing your data? Does it require a lot of state?

For example, this query:

SELECT
  user_id,
  count(*)
FROM user_logins

will need a state proportional with the number of users.

You can tune the memory of TM in the options.

Here is a useful link: https://www.ververica.com/blog/how-to-size-your-apache-flink-cluster-general-guidelines

Concurrent jobs: is this machine under-used, and do you need to keep a pool of unused TS ready to execute a job?

A TM's memory will be sliced between the TS (be sure it fits your state size), but the CPU will be shared when idle.

Other than that if it's going fine on one TM on one pod then you have nothing to do.

Upvotes: 0

Related Questions