CODEWITHSUNDEEP
scalaapache-sparkmergetypesafe
Ferdi777
Ferdi777

Reputation: 347

Merging configurations for spark using typesafe library and extraJavaOptions

I'm trying to merge 2 config file (or create a config file based on a single reference file) using 

  lazy val finalConfig:
    Option(System.getProperty("user.resource"))
      .map(ConfigFactory.load)
      .map(_.withFallback(ConfigFactory.load(System.getProperty("config.resource"))).resolve())
      .getOrElse(ConfigFactory.load(System.getProperty("config.resource")))

I'm defining my java variable inside spark using spark-submit ....... --conf spark.driver.extraJavaOptions=-Dconfig.resource=./reference.conf,-Duser.resource=./user.conf ...

My goal is to be able to point a file that is not inside my jar to be used by System.getProperty("..") in my code. I changed the folder for testing (cd ..) and keep getting the same error so I guess spark doesn't care about my java arguments..?

Is there a way to point to a file (or even 2 files in my case) so that they can be merged?

I also tried to include the reference.conf file but not the user.conf file: it recognizes the reference.conf but not the user.conf that i gave with --conf spark.driver.extraJavaOptions=-Duser.resource=./user.conf .

Is there a way to do that? Thanks if you can help

Upvotes: 2

Views: 611

Answers (1)

Salim
Salim

Reputation: 2178

I don't see you doing ConfigFactory.parseFile to loaded a file containing properties.

Typesafe automatically read any .properties file in the class path, all -D parameters passed in to the JVM and then merges them.

I am reading an external property file which is not part of the jar as following. The file "application.conf" is placed on the same directory where the jar is kept.

    val applicationRootPath = System.getProperty("user.dir")
    val config = Try {
      ConfigFactory.parseFile(new File(applicationRootPath + "/" + "application.conf"))
    }.getOrElse(ConfigFactory.empty())

    appConfig = config.withFallback(ConfigFactory.load()).resolve

ConfigFactory.load() already contains all the properties present on the properties files in the class path and -d parameters. I am giving priority to my external "application.conf" and falling back on default values. For matching keys "application.conf" take precedence over other sources.

Upvotes: 1

Related Questions