CODEWITHSUNDEEP
scalaapache-spark
wili
wili

Reputation: 123

SparkException: Cannot load main class from JAR file:/root/master

I want to use spark-submit to submit my spark application. The version of spark is 2.4.3. I can run the application by java -jar scala.jar.But there has some error when I run spark-submit master local --class HelloWorld scala.jar.

I am trying to change the submit-method including local, spark://ip:port but has not result. there is always throwing the error below when I modify path of jar anyway.

There is the code of my application.

import org.apache.spark.{SparkConf, SparkContext}

object HelloWorld {
  def main(args: Array[String]): Unit = {
    println("begin~!")
    def conf = new SparkConf().setAppName("first").setMaster("local")
    def sc = new SparkContext(conf)
    def rdd = sc.parallelize(Array(1,2,3))

    println(rdd.count())
    println("Hello World")
    sc.stop()
  }
}

When I use spark-submit the error below will happen.

Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file:/root/master
    at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
    at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:221)
    at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116)
    at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:911)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:911)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Upvotes: 2

Views: 6490

Answers (1)

wili
wili

Reputation: 123

I am very sorry, the reason of the error happens is I forget to add '--' before master. So I try to run application by spark-submit --master local --class HelloWorld scala.jar. Finally, it is work fine.

Upvotes: 3

Related Questions