CODEWITHSUNDEEP
apache-sparkapache-spark-sql
wazza
wazza

Reputation: 800

How to calculate number of executors and memory for each?

Hi I have two nodes of each 16 GB RAM and 4 cores. Can any one suggest me spark properties such as no of executors and executor memory to use it effectively?

I am using spark sql query (Select * query with 2 joins) and also suggest me a way so that I want to bring down this querying process time to 1 seconds and my input data 10 GB as of now

Upvotes: 2

Views: 3639

Answers (2)

Shahzad Aslam
Shahzad Aslam

Reputation: 300

There are two ways to allocate resources Static and Dynamic allocation. Here is how you can allocate static resources to you applications

Your total cluster resources 16 GB RAM 4 cores

First 1 core and 1 GB should be allocated for the OS leaving 15 GB RAM and 3 Cores

Core is the concurrency level in Spark so as you have 3 cores you can have 3 concurrent processes running simultaneously.

Next come the calculation for the number of executors. Each executor run in its own JVM process and each Worker node can run multiple executors. This is recommended that each executor should have at most 5 concurrent processes otherwise contention amount concurrent processes will be not be manageable in a single JVM instance. As you have only 3 cores so you can have 1 executor

dividing 15 GB of RAM in 3 cores will leave 5 GB for each core.

So in your case the you will set following properties 

Number of cores = 3
Number of Executors = 1
RAM = 15 GB

Upvotes: 0

WestCoastProjects
WestCoastProjects

Reputation: 63022

Typical workloads may use in the range of 2 to 8GB per process. Given your small resources then maybe 3G per executor and 4 executors (one per core) . That leaves plenty of ram (4gb) for the O/S and other processes.

Using 4 executors would be the default for spark standalone (you are using that or are you using Yarn?) In that case you do not need to specify it explicitly. But just in case:

Standalone (and mesos):

 --total-executor-cores=8

Yarn:

 --num-executors=1 --executor-cores=4

The other setting is

 --executor-memory=3G

Upvotes: 3

Related Questions