CODEWITHSUNDEEP
apache-sparkpysparkamazon-emr
user7343922
user7343922

Reputation: 306

spark: How reducing executor cores solve memory issue?

When I was searching for a memory related issue in spark, I came across this article, which is suggesting to reduce the number of cores per executor, but in the same article it's mentioned that we have get the number of executors using the formula ((number of cores per node * total no of nodes)/no of cores per executor), so if we reduce the number of cores per executor, the number of executor increases. So how will we solve the problem by reducing the number of cores per executor?

Upvotes: 0

Views: 674

Answers (1)

Young
Young

Reputation: 584

In fact the optimization mentioned in this article is pure theory:

first he implicitly supposed that the number of executors doesn't change even when he reduces the cores per executor from 5 to 4. cuz normally when we change the cores per executor, the number of executors could change since nb executor = nb core / excutor cores and the nb of cores fixed in your cluster

but in his case, the nb of executor is always 3, so the ram per executor keeps as 36 GB, which allows more ram per core(ram per core = ram per executor / cores per executor) 36GB/4 = 9Gb which is bigger than 36GB / 5 = 7GB.

Secondly he supposed that the tasks requires more than 7 GB but less than 9 GB which for me cannot be predicted beforehand. the data required per task depends on the input data and the data distribution and cannot be easily controlled in a precise range.

I prefer to say that, reducing disk spill during the spark job is a good way to optimize your spark performance, reducing executor core is one of the methods that might reduce disk spill(however I am sure it's not always the effective way)

Upvotes: 1

Related Questions