CODEWITHSUNDEEP
apache-sparkpyspark
ml_0x
ml_0x

Reputation: 322

Spark: Optimal use of executors

I am running a Spark application with 5 executors with 5 cores per executor. However, I have noticed that only a single executor does most of the work (i.e most of the tasks are done there). The jobs that I am running are highly parallel (20 partitions or greater). How do you explain this behavior?

Even if I decrease the number of cores per executor, results to just running less tasks on that single executor at the same time. Should I limit the memory per executor so that more executors are used (just in case the whole data fits on a single executor)?

Upvotes: 0

Views: 873

Answers (1)

anwartheravian
anwartheravian

Reputation: 1101

Just to add my two cents, for people facing this issue in future. This kind of issue usually arises as a result of skewed partition size of the RDD/Dataframe. To debug the problem, you can check the partition sizes of the RDD, to find out if there is any outlier there. And if there is one, you can see the elements in that big partition to get a sense of what's going on.

A similar issue is addressed in detail in this stackoverflow question.

Upvotes: 0

Related Questions