CODEWITHSUNDEEP
scalaapache-spark
monster
monster

Reputation: 1782

Passing Arguments in Apache Spark

I am running this code on a local machine: 

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/Users/username/Spark/README.md"
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }
}

I'd like to run the program but run it on different files - it currently only runs on README.md. How do I pass the file path of another file when running Spark (or any other argument for that matter?). For example, I'd like to change contains("a") to another letter.

I make the program run by:

$ YOUR_SPARK_HOME/bin/spark-submit \
  --class "SimpleApp" \
  --master local[4] \
  target/scala-2.10/simple-project_2.10-1.0.jar

Thanks!

Upvotes: 18

Views: 26162

Answers (2)

Spidey Praful
Spidey Praful

Reputation: 59

replace value of logFile variable with below

val logFile= args(0)

And, pass the actual value in an argument while running spark-submit like below-

spark-submit --master local --class "SimpleApp" target/scala-2.10/simpleapp_2.10-1.0.jar "/Users/username/Spark/README.md"

Upvotes: 0

suiterdev
suiterdev

Reputation: 691

When you set up your main in 

 def main(args: Array[String]) {

you are preparing your main to accept anything after the .jar line as an argument. It will make an array named 'args' for you out of them. You then access them as usual with args[n].

It might be good to check your arguments for type and/or format, it usually is if anyone other than you might run this.

So instead of setting the

val logFile = "String here"

set it

val logFile = args(0)

and then pass the file as the first argument. Check spark-submit docs for more on that, but, you just enter it on the next line basically.

Upvotes: 22

Related Questions