CODEWITHSUNDEEP
apache-spark
elgoog
elgoog

Reputation: 509

Running Spark on heterogeneous cluster in standalone mode

I have a cluster of 3 nodes, each has 12 cores, and 30G, 20G and 10G of RAM respectively. When I run my application, I set the executor memory to 20G, which prevent the executor from being launched in the 10G machine since it's exceeding the slave memory threshold, it also under utilize the resources on the 30G machine. I searched but didn't find any way to set the executor memory dynamically base on the capacity of the node, so how can I config the cluster or my Spark job to fully utilize the resources of the cluster?

Upvotes: 1

Views: 596

Answers (1)

David
David

Reputation: 11593

The solution is to have more executors with to less memory. You can use all of the memory by having 6- 10G executors (1 on the 10G node, 2 on the 20G node, 3 on the 30G node). Or by having 12- 5G executors. etc

Upvotes: 1

Related Questions