CODEWITHSUNDEEP
apache-sparkhadoop-yarn
Santosh Kumar
Santosh Kumar

Reputation: 791

Spark executor on yarn-client does not take executor core count configuration.

Irrespective of the spark executor core count, yarn container for the executor does not use more than 1 core.

Upvotes: 6

Views: 2655

Answers (1)

banjara
banjara

Reputation: 3890

YARN is showing 1 core per executor irrespective of spark.executor.cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

Use DominantResourceCalculator, It uses both cpu and memory.

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

More about DominantResourceCalculator

Upvotes: 11

Related Questions