CODEWITHSUNDEEP
apache-sparkhadoopgoogle-cloud-dataproc
Triffids
Triffids

Reputation: 155

Spark thrift server use only 2 cores

Google dataproc one node cluster, VCores Total = 8. I've tried from user spark:

/usr/lib/spark/sbin/start-thriftserver.sh --num-executors 2 --executor-cores 4

tried to change /usr/lib/spark/conf/spark-defaults.conf

tried to execute

   export SPARK_WORKER_INSTANCES=6
   export SPARK_WORKER_CORES=8

before start-thriftserver.sh

No success. In yarn UI I can see that thrift app use only 2 cores and 6 cores available.

UPDATE1: environment tab at spark ui:

spark.submit.deployMode client
spark.master    yarn
spark.dynamicAllocation.minExecutors    6
spark.dynamicAllocation.maxExecutors    10000
spark.executor.cores    4
spark.executor.instances    1

yarn ui spark ui

Upvotes: 0

Views: 1497

Answers (2)

VitaliyD
VitaliyD

Reputation: 81

If you want YARN shows you proper number of cores allocated to executors change value in capacity-scheduler.xml for: 

yarn.scheduler.capacity.resource-calculator

from: 

org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator

to:

org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

Otherwise it doesn't matter how many cores you ask for your executors, YARN will show you only one core per container.

Actually this config changes resource allocation behavior. More details: https://hortonworks.com/blog/managing-cpu-resources-in-your-hadoop-yarn-clusters/

Upvotes: 0

Artur Sukhenko
Artur Sukhenko

Reputation: 652

It depends on what yarn mode is that app in. Can be yarn client - 1 core for Application Master(the app will be running on the machine where you ran command start-thriftserver.sh). In case of yarn cluster - Driver will be inside AM container, so you can tweak cores with spark.driver.cores. Other cores will be used by executors (1 executor = 1 core by default) Beware that --num-executors 2 --executor-cores 4 wouldn't work as you have 8 cores max and +1 will be needed for AM container (total of 9) You can check cores usage from Spark UI - http://sparkhistoryserverip:18080/history/application_1534847473069_0001/executors/

Options below are only for Spark standalone mode:

export SPARK_WORKER_INSTANCES=6
export SPARK_WORKER_CORES=8

Please review all configs here - Spark Configuration (latest)

In your case you can edit spark-defaults.conf and add:

spark.executor.cores 3
spark.executor.instances 2

Or use local[8] mode as you have only one node anyway.

Upvotes: 3

Related Questions