CODEWITHSUNDEEP
apache-spark
sakhdeek
sakhdeek

Reputation: 25

Whats the command to get spark driver memory in spark shell

I know spark related configuration can be get via spark-env.sh file however what would be the command to get it from spark-shell ? For example to get spark.driver.memory shall I use 

set spark.driver.memory

above isn't working

Upvotes: 0

Views: 886

Answers (1)

hp2326
hp2326

Reputation: 181

You can provide the memory as a configuration while launching spark-shell

spark-shell --conf spark.driver.memory=2g

This will start a spark shell with 2g of driver memory. In order to access it in spark shell, you can do the following.

val conf = sparkContext.getConf
val driverMemory = conf.get("spark.driver.memory")

This will return String = 2g.

Upvotes: 1

Related Questions