CODEWITHSUNDEEP
amazon-web-servicesapache-sparkpyspark
thentangler
thentangler

Reputation: 1256

Setting spark.driver.maxResultSize in EMR

I am trying to set the spark.driver.maxResultSize in my pyspark job. I tried setting the conf setting inside my pyspark script like so:

spark.conf.set("spark.driver.maxResultSize", "4g")

But in my Spark environment it still shows the default 2G

enter image description here

I tried setting it in my AWS EMR spark-submit options with both --spark.driver.maxResultSize 4g and --conf spark.driver.maxResultSize 4g I get unrecongnized option as an error.

Am I missing the correct syntax?

Upvotes: 0

Views: 1278

Answers (1)

thentangler
thentangler

Reputation: 1256

I guess the syntax is --conf spark.driver.maxResultSize=4gnot --conf spark.driver.maxResultSize 4g That seems to work and is showing up correctly in the Spark Environment.

Upvotes: 1

Related Questions