Optimal settings for apache spark based on the hardware

Question

is there a mapping/translation for the number of hardware systems, cpu cores and their associated memory to the spark-submit tunables of: executor-memory executor-cores num-executors The application is certaionly bound to have something to do with these tunables, I am however looking for a "basic rule of thumb" Apache spark is running on yarn with hdfs in cluster mode. Not all the hardware systems in the spark/hadoop yarn cluster have the same number of cpu cores or RAM.

Answer 1

There is no thumb rule, but after considering

1. off heap memory
2. Number of applications and other hadoop dameons running
3. Resource manager needs
4. HDFS IO

etc.

You can derive a suitable configuration. Please check this url