CODEWITHSUNDEEP
scalaapache-spark
saurabht
saurabht

Reputation: 51

Reading a config file which is placed on my Linux node in scala

I am trying to read a config file using typesafe config library in Scala but I cannot place the conf file in my resource folder.

my property/config file is in below format

region=dev
numlines=2

and the name of the file is property.txt

the code looks as below

import com.typesafe.config._
val propertyFile = args(2)

val myConfigFile = new File(propertyFile)

val fileConfig = ConfigFactory.parseFile(myConfigFile)
val config = ConfigFactory.load(fileConfig)

val environment = config.getString("region")
val numberOfLinesToBeRemoved = config.getInt("numlines")

Upvotes: 1

Views: 2850

Answers (2)

Dinesh Shinkar
Dinesh Shinkar

Reputation: 129

We can pass and load external configuration file by passing jar argument in spark submit command as well.(No need to copy config file in jar resources folder but need to copy on each node in cluster on same path)

    1] Spark command:
spark/bin>spark-submit --class test.Example Test.jar -appconfig /home/folderpath/APP.conf

    2] Example.scala:
    import com.typesafe.config._
    import java.nio.file.Paths
    ...........
    def main(argument: Array[String]) {
    var confPath = ""
    var argumentIndex = 0
    for (n <- 0 until argument.length) {
           if (argument(n).equals("-appconfig")) {
            argumentIndex = n + 1;
            confPath = argument(argumentIndex)
          }
    }
    val confFile = Paths.get(confPath).toFile
    val appConf = ConfigFactory.parseFile(confFile)
    name = appConf.getString("NAME")
    println("name : "+name)
    }

Upvotes: 0

sarveshseri
sarveshseri

Reputation: 13985

Case 1 - Lets say, you have a sbt project then your config file abc.conf should be located at src/main/resources/abc.conf.

Now lets say that the file abc.conf has following content.

region=dev
numlines=2

Now, you can access these configs by,

import com.typesafe.config._

val confFileName = "abc"

val config = ConfigFactory.load(confFileName)

val environment = config.getString("region")
val numberOfLinesToBeRemoved = config.getInt("numlines")

Case 2 - If you can not include the conf file as a resource in your project, then you can use pass the conf file path as an argument to java command.

import com.typesafe.config._

val config = ConfigFactory.load()

val environment = config.getString("region")
val numberOfLinesToBeRemoved = config.getInt("numlines")

Now, you will have to pass the config file path while running the application as,

java -jar your_jar.jar -Dconfig.file=path/to/config-file

Case 3 - You want to use the config from some specified path

import com.typesafe.config._
import java.nio.file.Paths

// It should be absolute or full path
val confFilePath = "/home/your_username/something/abc.conf"

// just replace the above line by
// val confFilePath = args(2)
// and pass the full path as command line argument.


val confFile = Paths.get(confFilePath).toFile

val config = ConfigFactory.parseFile(confFile)

val environment = config.getString("region")
val numberOfLinesToBeRemoved = config.getInt("numlines")

Upvotes: 6

Related Questions