CODEWITHSUNDEEP
apache-flinkflink-streaming
Folani
Folani

Reputation: 51

Determination of the location of a specific operator in Flink

Assume that I have a cluster of machines with different powers (i.e. different CPU, Memory, etc.). Is there any way to determine the location of a specific operator (or maybe task) to run? Is it possible to change the default running location of an operator?

I read about "Task chaining and resource groups". However, I think it is not solve my problem.

Is there any specific configuration in Flink? if no, What part of code may relates to this assumption?

Upvotes: 1

Views: 207

Answers (2)

Sandesh Mendon
Sandesh Mendon

Reputation: 9

Each bunch of task can be tagged to some name and id as below:

inputStream
 .map()
 .keyBy()
 .window()
 .reduce()
 .name("enriching customer orders info")
 .uid("enriching customer orders info")

this way each operator can be identified uniquely you can see these names in DAG

hope this helps.

Upvotes: 0

CantankerousBullMoose
CantankerousBullMoose

Reputation: 512

The taskmanager.numberOfTaskSlots property can be an avenue to indicating the relative working capacity of different taskmanagers. You would likely have to adjust both the taskSlots and the memory allocation options based on the resources available resources on the machine.

Personally though, I prefer to address this type of problem with containers, like docker. That way, you can have all of your taskmanagers configure the same, and let the orchestration tools in your container system worry about assigning more or fewer virtual instances to a physical machine base on available resources.

Upvotes: 1

Related Questions