CODEWITHSUNDEEP
javamavenapache-spark
jane
jane

Reputation: 177

spark maven jar file dependency

I have used Eclipse with Maven, Java and Scala to develop a simple Spark application. Under the target folder, Maven install command has created two JAR files one named as application.jar and the other one named as application-with-dependencies.jar. My question is which JAR file I should use to submit it to the Spark cluster? Also, why there are two JAR files? The file called application.jar is fraction of the size application-with-dependencies.jar

Upvotes: 1

Views: 1299

Answers (1)

Farrandu
Farrandu

Reputation: 371

Maven usually generates a JAR file containing only your compiled classes and resources, but using some plugins, like maven-assembly-plugin, maven can also generate an uber-jar, i.e.: a JAR file that contains not only your classes and resources, but also libraries your project depends on.

Without more information application-with-dependencies-jar looks like one of those uber-jar.

From spark documentation looks like that's the JAR you must submit to the Spark cluster.

Upvotes: 2

Related Questions