CODEWITHSUNDEEP
scalaapache-sparktypesafe-config
Antonio Cachuan
Antonio Cachuan

Reputation: 485

Submit an application property file with Spark typesafe config

Please, I need your help, I'm trying to submit an external configuration file for my spark application using typesafe config.

I'm loading the application.conf file in my application code like this:

  lazy val conf = ConfigFactory.load()

File content

  ingestion{
  process {
    value = "sas"
  }
  sas {
    origin{
      value = "/route"
    }
    destination{
      value = "/route"
    }
    extension{
      value = ".sas7bdat"
    }
    file{
      value = "mytable"
    }
    month{
      value = "201010,201011"
    }
    table{
      value = "tbl"
    }
  }
}

My spark submit is

spark2-submit --class com.antonio.Main --master yarn --deploy-mode client --driver-memory 10G --driver-cores 8 --executor-memory 13G --executor-cores 4 --num-executors 10 --verbose  --files properties.conf /home/user/ingestion-1.0-SNAPSHOT-jar-with-dependencies.jar --files application.conf

But for some reason, I'm receiving

com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'ingestion'

Everything looks configured correctly ?? Have I missed something.

thanks,

Antonio

Upvotes: 1

Views: 5499

Answers (1)

Vladimir Matveev
Vladimir Matveev

Reputation: 128031

Your application.conf by default must be present at the root of classpath for ConfigFactory.load() to find it. Alternatively, you can modify where to find the application.conf file through system properties. Therefore, your options are as follows.

First alternative is, add the root directory of the job to classpath:

spark2-submit ... \
    --conf spark.driver.extraClassPath=./  \
    --conf spark.executor.extraClassPath=./  \    // if you need to load config at executors
    ...

Keep the --files option as is. Note that if you run your job in the client mode, you must pass the proper path to where application.conf is located on the driver machine to the spark.driver.extraClassPath option.

Second alternative is (and I think this one is superior), you can use the config.file system property to affect where ConfigFactory.load() looks for the config file:

spark2-submit ... \
    --conf spark.driver.extraJavaOptions=-Dconfig.file=./application.conf \
    --conf spark.executor.extraJavaOptions=-Dconfig.file=./application.conf \
    ...

Remarks about loading config on executors and keeping the --files option also apply here.

Upvotes: 6

Related Questions