CODEWITHSUNDEEP
apache-spark
SChorlton
SChorlton

Reputation: 123

Spark --num-executors and number of partitions

Do the number of executors have to be a factor of the partition count. As I am getting data loss when having 2 executors and 25 partitions.

Upvotes: 3

Views: 3163

Answers (1)

Vince.Bdn
Vince.Bdn

Reputation: 1175

Not at all! The number of partitions is totally independent from the number of executors (though for performance you should at least set your number of partitions as the number of cores per executor times the number of executors so that you can use full parallelism!).

Maybe you can post your code so that we can tell why you have data loss...

Upvotes: 3

Related Questions