CODEWITHSUNDEEP
hadoopapache-spark
CatNinja
CatNinja

Reputation: 109

Override core-site.xml property value at spark-submit execution time

Is there a way to override core-site property value when executing spark submit?

I can use HADOOP_CONF_DIR variable in spark-env.sh to point to a new core-site file, but I only want to override few values to to use different storage accounts for each spark job.

Upvotes: 2

Views: 2865

Answers (1)

CatNinja
CatNinja

Reputation: 109

Found answer to my own question.

hadoop-related configuration can be overridden by pre-fixing "spark.hadoop" to property key, then submit to sparkconf.

i.e. spark-submit --sparkconf spark.hadoop.io.file.buffer.size 12952

See source code here: https://github.com/apache/spark/commit/b6cf1348170951396a6a5d8a65fb670382304f5b

Upvotes: 3

Related Questions