CODEWITHSUNDEEP
apache-spark
prady
prady

Reputation: 593

How many executors required to process the 80 GB file?

Available Hardware Config - 16 GB with 8 Cores

What will be my number of executors and executor memory required to process the data ?

Data format - data is delimited text format with 20 columns , aggregation like average needs to be performed on one of the column .

Upvotes: 0

Views: 1137

Answers (1)

Kirk Broadhurst
Kirk Broadhurst

Reputation: 28728

More executors only makes a job or stage complete faster.

Your work will be broken up into tasks, and each of those tasks will be run by an executor. If there are more tasks than executors, they simply queue. If you have only one executor then the queue is longer! This should determine the ability to complete.

A better question would be 'how much memory should I give to an executor'.

Upvotes: 1

Related Questions