CODEWITHSUNDEEP
apache-spark
Srinivas
Srinivas

Reputation: 2100

Why does spark-submit fail with "IllegalArgumentException: Missing application resource."?

When I create the jar of my Spark application and try to run it using spark-submit, I am getting the following error.

This is the command I used to run. 

spark-submit --executor-memory 1g --jars s3://test-data-lab-users/spachari/test/test_2.10-1.0.jar

This is the error i am getting. Does this mean I have not passed correct parameters in my spark-submit?

Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)
Command exiting with ret '1'

Upvotes: 18

Views: 40360

Answers (3)

abdul hafiz
abdul hafiz

Reputation: 1

This is because main Jar is missing in spark-submit. Check at --class argument. It should be like:

--class "your class name with package" "main jar path" "jar parameters"

Upvotes: 0

Aman Sehgal
Aman Sehgal

Reputation: 556

--jars option is added when you've to supply supporting jars to your application jar.

Application resource missing implies that your main jar is missing as you've passed it with --jars option. It's looking for your main jar and since it couldn't find it, it throws that error.

Upvotes: 4

Jacek Laskowski
Jacek Laskowski

Reputation: 74669

tl;dr Remove --jars option and start over.

java.lang.IllegalArgumentException: Missing application resource.

You missed your...well...Spark application that the message refers to as "application resource".

That's more obvious when you execute spark-submit and see the different command-line options and their meanings.

./bin/spark-submit
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]

That part <app jar | python file | R file> is what you missed.

To reproduce your issue you can simply execute spark-submit with --jars options without specifying the main jar or class of a Spark application.

$ ./bin/spark-submit --jars target/spark-parent_2.11-2.3.0-SNAPSHOT-tests.jar
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:274)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)

Quoting spark-submit --help, --jars is...

--jars JARS Comma-separated list of jars to include on the driver and executor classpaths.

--jars can be very helpful when a Spark application depends on additional jar files (aka dependencies), i.e. mysql-connect.jar that you cannot (or most likely don't want to) "assembly" to your uber jar.

Upvotes: 17

Related Questions