CODEWITHSUNDEEP
apache-flink
Cheng Jiang
Cheng Jiang

Reputation: 225

Apache Flink: number of TaskManagers per machine

The number of CPU cores per machine is four. In flink standalone mode, how should I set the number of TaskManagers on each machine?

  1. 1 TaskManager, each TaskManager has 4 slots.
  2. 2 TaskManagers, each TaskManager has 2 slots.
  3. 4 TaskManagers, each TaskManager has 1 slot. This setting is like apache-storm.

Upvotes: 1

Views: 2013

Answers (3)

John Seekins
John Seekins

Reputation: 21

There's also the consideration of Flink's scheduling algorithm. We've frequently run into problems where, with multiple hosts running one large task manager a piece, all jobs get scheduled to one host, which can cause load problems.

We ended up making multiple smaller task managers per host and jobs seem to be distributed better (although they still cluster on one node often).

So, in my experience, I'd lean more towards 4 task managers with 1 slot a piece, or maybe compromise at 2 task managers with 2 slots a piece.

Upvotes: 2

kkrugler
kkrugler

Reputation: 9245

Normally you'd have one TaskManager per server, and (as per the doc that bupt_ljy referenced) one slot per physical CPU core. So I'd go with your option #1.

Upvotes: 2

Jiayi Liao
Jiayi Liao

Reputation: 1009

I think it depends on your application.

In official documents Distributed Runtime Environment, it says As a rule-of-thumb, a good default number of task slots would be the number of CPU cores. With hyper-threading, each slot then takes 2 or more hardware thread contexts.

But if you have to use a lot of memory in your application, then you don't need too many slots in one task manager.

Upvotes: 0

Related Questions