CODEWITHSUNDEEP
scalaamazon-web-servicesapache-spark
Iyer
Iyer

Reputation: 41

A master url must be set to your configuration (Spark scala on AWS)

This is what I ran (in scala spark)

This is what I wrote via intellij. I plan on eventually writing larger spark scala files.

Anyways, I uploaded it on an AWS cluster that I had made. The "master" line, line 11 was "master("local")". I ran into this error

The second picture is the error that was returned by AWS when it did not run successfully. i changed line 11 to "yarn" instead of local (see the first picture for its current state)

It still is returning the same error. I put in the following flags when I uploaded it manually --steps Type=CUSTOM_JAR,Name="SimpleApp"

It worked two weeks ago. My friend did almost the exact same thing as me. I am not sure why it isn't working.

I am looking for both a brief explanation and an answer. Looks like I need a little more knowledge on how spark works.

I am working with amazon EMR. the error AWS gave

Upvotes: 1

Views: 3256

Answers (2)

Naseer khan
Naseer khan

Reputation: 54

Try using the below code:

val spark = SparkSession.builder().master("spark://ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com:xxxx").appName("example").getOrCreate()

you need to provide the proper link to your aws cluster.

Upvotes: 1

VladoDemcak
VladoDemcak

Reputation: 5259

I think on the line 9 you are creating SparkContext with "old way" approach in spark 1.6.x and older version - you need to set master in default configuration file (usually location conf/spark-defaults.conf) or pass it to spark-submit (it is required in new SparkConf())...

On line 10 you are creating "spark" context with SparkSesion which is approach in spark 2.0.0. So in my opinion your problem is line num. 9 and I think you should remove it and work with SparkSesion or set reqiered configuration for SparkContext In case when you need sc.

You can access to sparkContext with sparkSession.sparkContext();

If you still want to use SparkConf you need to define master programatically:

val sparkConf = new SparkConf()
                    .setAppName("spark-application-name")
                    .setMaster("local[4]")
                    .set("spark.executor.memory","512m");

or with declarative approach in conf/spark-defaults.conf

spark.master            local[4]
spark.executor.memory   512m

or simply at runtime:

./bin/spark-submit --name "spark-application-name" --master local[4] --executor-memory 512m your-spark-job.jar

Upvotes: 1

Related Questions