CODEWITHSUNDEEP
parallel-processingapache-flinkflink-streaming
Jerome Li
Jerome Li

Reputation: 176

Flink CPU configuration

Official Flink Documentation states that for each core in your cpu, you have to allocate 1 slot and increase the parallelism level by one simultaneously.

One of my custom operators requires more than 1 CPU for computing (It is how it works in Heron). My system's cpu is 2.5. But Flink only uses 1 cpu. Do you know how can I config the Flink to use more CPU with only 1 slot?

Upvotes: 0

Views: 482

Answers (2)

Litchy
Litchy

Reputation: 673

You could use one slot, but a multi-thread operation in your flink code

for example in scala,

env.setParallelism(1)
class myMap extends RichMapFunction {
  override def map(...) = {
    data.toParArray()... // this is scala concurrent collection, other language has similar ones, like c/cpp OMP, java multi-thread
  }
}

then the flink operator has parallelism of 1, but the internal parallelism is controlled by your map method.

Upvotes: 0

David Anderson
David Anderson

Reputation: 43707

"One slot per core" is merely a rule of thumb. Nothing enforces this.

Each subtask (an instance of an operator chain) is single-threaded, but the slots within a task manager, and the task managers within a machine or container, will use all of the resources made available to them.

Upvotes: 0

Related Questions