CODEWITHSUNDEEP
scalaapache-spark
Surender Raja
Surender Raja

Reputation: 3599

Not able to read conf file in spark scala

I would like to read a conf file in to my spark application. The conf file is located in Hadoop edge node directory.

omega.conf 

username = "surrender"
location = "USA"

My Spark Code :

package com.test.spark

import org.apache.spark.{SparkConf, SparkContext}
import java.io.File
import com.typesafe.config.{ Config, ConfigFactory }


object DemoMain {

 def main(args: Array[String]): Unit = {
 println("Lets Get Started ")
 val conf = new SparkConf().setAppName("SIMPLE")
 val sc = new SparkContext(conf)
 val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
 val conf_loc = "/home/cloudera/localinputfiles/omega.conf"
 loadConfigFile(conf_loc)

 }

   def loadConfigFile(loc:String):Unit ={
       val config = ConfigFactory.parseFile(new File(loc))
       val username = config.getString("username")
       println(username)

 }

}

I am running this spark application using spark-submit 

  spark-submit --class com.test.spark.DemoMain --master local /home/cloudera/dev/jars/spark_examples.jar

Spark job is initiated ,but it throws me the below error .It says that No configuration setting found for key 'username'

   17/03/29 12:57:37 INFO SparkContext: Created broadcast 0 from textFile at DemoMain.scala:25
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'username'
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
    at com.typesafe.config.impl.SimpleConfig.getString  (SimpleConfig.java:197)
    at com.test.spark.DemoMain$.loadConfigFile(DemoMain.scala:53)
    at com.test.spark.DemoMain$.main(DemoMain.scala:27)
    at com.test.spark.DemoMain.main(DemoMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at  org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Please help me on fixing this issue

Upvotes: 2

Views: 4537

Answers (1)

Sandeep Purohit
Sandeep Purohit

Reputation: 3692

I just tried its working fine i test this with below code

val config=ConfigFactory.parseFile(new File("/home/sandy/my.conf"))
  println("::::::::::::::::::::"+config.getString("username"))

and conf file is

username = "surrender"

location = "USA"

Please check location of your file by printing it.

Upvotes: 4

Related Questions