CODEWITHSUNDEEP
scalaapache-sparkhoconscopt
Egyptian
Egyptian

Reputation: 63

spark submit-read command line argument which is config file [HOCON] using Scopt and ConfigFactory

I have a jar that I am running through spark-submit and earlier i was using Argot to parse a HOCON config file which was parsed through ConfigFactory and then I'd read from there.

 * spark/bin/spark-submit --class ConsumerApp \
 *                        --master local[2] \
 *                        some-consumer-jar-0.1.0.jar \
 *                        --config config.hocon

Unfortunately Argot is a dead project now and to upgrade to current version of Scala I have to start using Scopt , but I have trouble understanding how can I parse the same config file using Scopt and load in the ConfigFactory , without making too many changes.

Tried reading through documentation but it couldn't find much.

current implementation 

def main(args: Array[String]) {

    // General bumf for our app
    val parser = new ArgotParser(
      programName = "generated",
      compactUsage = true,
      preUsage = Some("%s: Version %s, %s.".format(
        generated.Settings.name,
        generated.Settings.version,
        generated.Settings.organization)
      )
    )

    // Optional config argument
    val config = parser.option[Config](List("config"),
      "filename",
      "Configuration file.") {
      (c, opt) =>

        val file = new File(c)
        if (file.exists) {
          ConfigFactory.parseFile(file)
        } else {
          parser.usage("Configuration file \"%s\" does not exist".format(c))
          ConfigFactory.empty()
        }
    }
    parser.parse(args)

    // read the config file if --config parameter is provided else fail
    val conf = config.value.getOrElse(throw new RuntimeException("--config argument must be provided"))

and then read it like this 

arg1 = conf.getConfig("somelevel").getString("arg1"),

Upvotes: 0

Views: 783

Answers (0)

Related Questions