CODEWITHSUNDEEP
apache-sparkpyspark
Ego
Ego

Reputation: 585

Setting number of cores used in PySpark script?

I would like to know how to set the number of cores to be used in a PySpark program.

I have been doing a bit of searching and have been unable to find a definitive answer.

Upvotes: 2

Views: 7296

Answers (2)

None
None

Reputation: 1468

You can set it using --executor-cores with spark-submit or set it using sc.setExecutorEnv(key=None, value=None, pairs=None) in the code it self.

Upvotes: 2

user1261215
user1261215

Reputation:

You can use --executor-cores property to specify the number of cores to use while submitting application with spark-submit.

Below is an example:

./bin/spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn-cluster \
    --num-executors 3 \
    --driver-memory 4g \
    --executor-memory 2g \
    --executor-cores 1
    lib/spark-examples*.jar \
    10

Upvotes: 1

Related Questions