CODEWITHSUNDEEP
scalaapache-spark
Rohini Mathur
Rohini Mathur

Reputation: 441

Apache spark master URL error running in idea intellij

i am using intellij idea for spark application in scala. i am not sure why i am getting below error.

Code:

package batch

import java.lang.management.ManagementFactory

import org.apache.spark.{SparkConf, SparkContext}

object BatchJob {
  def main(args: Array[String]): Unit = {


    val conf = new SparkConf()
      .setAppName("Lambda with Spark")

    val sc = new SparkContext(conf)
    val sourceFile = "file:///E:/Boxes/data.tsv"
    val input = sc.textFile(sourceFile)

    input.foreach(println)
  }

}

Error:

**Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration**

Upvotes: 0

Views: 832

Answers (3)

Carlos Verdes
Carlos Verdes

Reputation: 3147

Previous answers force the driver to run in local with a number of cores. This is fine to do a quick test on local but is not good if you want to execute this driver with Spark submit command.

The trick that worked for me is to create a default conf and check if master is defined. If it's not then I assume is a local test and I force master = local.

val SPARK_MASTER = "spark.master"
val DEFAULT_MASTER = "local[*]"

// get default conf
val defaultConf = new SparkConf()

// if master is not defined then set default to local
if(!defaultConf.contains(SPARK_MASTER)) defaultConf.setMaster(DEFAULT_MASTER)

val sparkSession = SparkSession.builder().config(defaultConf).getOrCreate()

Upvotes: 1

Soheil Pourbafrani
Soheil Pourbafrani

Reputation: 3427

If you're running Spark on Intellij (in local mode), You should set master for the Spark config object, too:

val conf = new SparkConf().setAppName("Lambda with Spark").setMaster("local[*]")

Upvotes: 0

Kuldip Puri Tejaswi
Kuldip Puri Tejaswi

Reputation: 452

The problem is exactly what the error says ,u have to set a master url for spark to run.If you are running inside ide or locally you should make SparkConf object like this:

val conf = new SparkConf()
  .setAppName("Lambda with Spark").setMaster(local[*])

for running on clusters you can use 'yarn' as the master.

Upvotes: 0

Related Questions