CODEWITHSUNDEEP
apache-spark
David S.
David S.

Reputation: 11200

How to set Spark executor memory?

I have set spark.executor.memory to 2048m, and in the UI "Environment" page, I can see this value has been set correctly. But in the "Executors" page, I saw there's only 1 executor and its memory is 265.4MB. Very strange value. why not 256MB, or just as what I set?

What am I missing here?

Upvotes: 4

Views: 13451

Answers (2)

ᐅdevrimbaris
ᐅdevrimbaris

Reputation: 796

Please see the following question for the 265.4MB memory size...

How to set Apache Spark Executor memory

Upvotes: 2

Daniel Darabos
Daniel Darabos

Reputation: 27470

The "Executors" tab on the UI also includes the driver in the list. Its "executor ID" is listed as <driver>. This process is not started by Spark, so it is not affected by spark.executor.memory.

  • If you start the driver with spark-submit, its maximal memory can be controlled by spark.driver.memory or --driver-memory
  • If you start it as a plain old Java program, use the usual -Xmx Java flag.

Upvotes: 6

Related Questions