CODEWITHSUNDEEP
apache-spark
fo_x86
fo_x86

Reputation: 2623

Can spark-defaults.conf resolve environment variables?

If I have a line like below in my spark-env.sh file

export MY_JARS==$(jars=(/my/lib/dir/*.jar); IFS=,; echo "${jars[*]}")

which gives me a comma delimited list of jars in /my/lib/dir, is there a way I can specify

spark.jars $MY_JARS

in the spark-defaults.conf?

Upvotes: 5

Views: 3211

Answers (3)

Devon Finninger
Devon Finninger

Reputation: 349

At least in Spark 3+, there is a way to do this: ${env:VAR_NAME}.

For instance if you want to add the current username to the Spark Metrics Namespace, add this to your spark-defaults.conf file:

spark.metrics.namespace=${env:USER}

The generated metrics will show the username instead of the default namespace:

 testuser.driver.BlockManager.disk.diskSpaceUsed_MB.csv
 testuser.driver.BlockManager.memory.maxMem_MB.csv
 testuser.driver.BlockManager.memory.maxOffHeapMem_MB.csv
 testuser.driver.BlockManager.memory.maxOnHeapMem_MB.csv
... etc ...

https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/internal/VariableSubstitution.html

A helper class that enables substitution using syntax like ${var}, ${system:var} and ${env:var}.

Upvotes: 3

Jacek Laskowski
Jacek Laskowski

Reputation: 74789

tl;dr No, it cannot, but there is a solution.

Spark reads the conf file as a properties file without any additional env var substitution.

What you could do however is to write the computed value MY_JARS from spark-env.sh straight to spark-defaults.conf using >> (append). The last wins so no worry there could be many similar entries.

Upvotes: 3

Sumit
Sumit

Reputation: 1420

I tried with Spark 1.4 and did not worked. spark-defaults.conf is a Key/ value and looking the code it seems values are not evaluated.

Upvotes: 0

Related Questions