CODEWITHSUNDEEP
apache-sparkhadoop-yarn
Kobe-Wan Kenobi
Kobe-Wan Kenobi

Reputation: 3874

Spark tuning job

I have a problem with tuning Spark jobs executing on Yarn cluster. I'm having a feeling that I'm not getting most of my cluster and additionally, my jobs fail (executors get removed all the time).

I have the following setup:

I have run my spark job (actually connected to a jupyter notebook) using different setups, e.g.

pyspark --master yarn --num-executors 7 --executor-cores 4 --executor-memory 3G

pyspark --master yarn --num-executors 7 --executor-cores 7 --executor-memory 2G

pyspark --master yarn --num-executors 11 --executor-cores 4 --executor-memory 1G

I've tried different combinations and none of them seems to be working as my executors get destroyed. Additionally, I've read somewhere that it is a good way to increase spark.yarn.executor.memoryOverhead to 600MB as a way not to loose executors (and I did that), but seems that doesn't help. How should I setup my job?

Additionally, it confuses me that when I look at the ResourceManager UI it says for my job vcores used 8 vcores total 56. It seems that I'm using a single core per executor, but I don't understand why?

One more thing, when I setup my job, how many partitions should I specify when I'm reading data from HDFS to get maximal performance?

Upvotes: 0

Views: 223

Answers (1)

Rohit Karlupia
Rohit Karlupia

Reputation: 166

Donald Knuth said premature optimisation is the root of all evil. I am sure faster running program which fails is on no use. Start by giving all the memory to one executor. Say 7GB/8GB and just 1 core. This is a complete wastage of cores, but if it works, it proves your application can possibly run on this hardware. If even this doesn't work, you should try getting bigger machines. Assuming it works, try increasing the number of cores, until it still works. The gist of the argument is: your application requires certain memory per task. But the number of tasks running per executor is dependent on number of cores. First find the worst case memory per cores for you application and then you can set executor memory and cores to some multiple of this number.

Upvotes: 0

Related Questions