CODEWITHSUNDEEP
javaapache-spark
us56
us56

Reputation: 327

java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession

I have written a Spark Job in Java. When I submit the Job it gives below error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
        at com.thinkbiganalytics.veon.util.SparkSessionBuilder.getOrCreateSparkSession(SparkSessionBuilder.java:12)
        at com.thinkbiganalytics.veon.AbstractSparkTransformation.initSparkSession(AbstractSparkTransformation.java:92)
        at com.thinkbiganalytics.veon.transformations.SDPServiceFeeDeductionSourceToEventStore.init(SDPServiceFeeDeductionSourceToEventStore.java:57)
        at com.thinkbiganalytics.veon.AbstractSparkTransformation.doTransform(AbstractSparkTransformation.java:51)
        at com.thinkbiganalytics.veon.transformations.SDPServiceFeeDeductionSourceToEventStore.main(SDPServiceFeeDeductionSourceToEventStore.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:745)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Upvotes: 29

Views: 67490

Answers (14)

vvthai10
vvthai10

Reputation: 101

In IntelliJ, update "Include dependencies with Provided scope" to "Add dependencies with provided scope to classpath"

Upvotes: 0

Sanjanaa Suresh
Sanjanaa Suresh

Reputation: 1

Solution

1. Update Maven pom.xml with maven-jar-plugin Configuration

<plugin>
    <!-- Build an executable JAR -->
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <archive>
            <manifest>
                <addClasspath>true</addClasspath>
                <classpathPrefix>lib/</classpathPrefix>
                <mainClass>com.spark.Main</mainClass>
            </manifest>
        </archive>      
    </configuration>
</plugin>

This configuration ensures that the generated JAR file includes the necessary dependencies and specifies the main class.

2. Run the JAR with Additional JVM Arguments

After building the JAR file, I ran it with the following additional JVM argument:

java -jar --add-opens=java.base/sun.nio.ch=ALL-UNNAMED my-application-jar-with-dependencies.jar

This argument ensures that the sun.nio.ch package is accessible to the unnamed module when running the application.

I hope this solution helps others facing similar issues!

Upvotes: 0

staticor
staticor

Reputation: 618

In my case downgrading the Java version from Java 18 to Java 8 fixed this issue. If someone knows why exactly the downgrade fixed the issue please comment and explain.

Upvotes: 0

Jayasurya T Appu
Jayasurya T Appu

Reputation: 1

If you are using Intellij, and had added a dependency in build.sbt do it like this:

[libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.4.1" % "provided" ]

If what I provided is mentioned then right click on the main file (scala object, scala class or Java) and click run <file name>, this will run the file and create a configuration. And in the configuration you can modify it and choose: add dependency with provided scope.

Upvotes: 0

Prem Prasun
Prem Prasun

Reputation: 21

Go to edit configuration and modify the configuration as per my screenshot:- enter image description here

Upvotes: 2

Chandan Gawri
Chandan Gawri

Reputation: 431

If you are running from IntelliJ, please check for "Include dependencies with Provided scope" as follows

Inside Run/Debug Configuration please select for Modify Options and then mark checked "Include dependencies with Provided scope"

Inside Run/Debug Configuration please select for Modify Options and then check

Upvotes: 17

user19912336
user19912336

Reputation: 1

I am using Maven project, got the same issue, I changed "provided" to "compile" in scope, pom.xml dependencies and made the same spark version which was installed in local spark shell. Problem got resolved.

Upvotes: 0

vagdevi k
vagdevi k

Reputation: 1685

If using Maven, go to your dependencies file(pom.xml) and change the scope from provided to compile.

<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-sql_2.13</artifactId>
   <version>3.3.0</version>
   <scope>compile</scope>
</dependency>

Upvotes: 2

ankursingh1000
ankursingh1000

Reputation: 1419

when submitting with spark-submit , check that your project has the same dependency as spark version in pom.xml,

This may be because you have two spark versions on the same machine

If you want to have different Spark installations on your machine, you can create different soft links and can use the exact spark version on which you have build your project

spark1-submit -> /Users/test/sparks/spark-1.6.2-bin-hadoop2.6/bin/spark-submit

spark2–submit -> /Users/test/sparks/spark-2.1.1-bin-hadoop2.7/bin/spark-submit

Here is a link from Cloudera blog about multiple Spark versions https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Multiple-Spark-version-on-the-same-cluster/td-p/39880

Upvotes: 10

Jeremy
Jeremy

Reputation: 1924

If you're running from inside Intellij IDEA, and you've marked your spark library as "provided", like so: "org.apache.spark" %% "spark-sql" % "3.0.1" % "provided", Then you need edit your Run/Debug configuration and check the "Include dependencies with Provided scope" box.

Upvotes: 37

userab
userab

Reputation: 573

I was facing this issue while running from the Intellij editor. I had marked the spark jars as provided in pom.xml, see below: 

<dependency>
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-sql_2.11</artifactId>
     <version>2.4.0</version>
     <scope>provided</scope>
 </dependency>

On removing the provided scope, the error was gone.

On making provided spark jars they would be provided only on running the application with spark-submit or having the spark jars on the classpath

Upvotes: 24

qian xi
qian xi

Reputation: 1

leaking of jars of spark environment will lead to this problem if you were using Intellij IDEA,you can following the steps below: File -> Project Structure -> Modules -> spark-examples_2.11 -> Dependencies jars -> {spark dir}/spark/assembly/target/scala-2.11/jars/

Upvotes: 0

T. Gawęda
T. Gawęda

Reputation: 16086

Probably you are deploying your application on the cluster with lower Spark version.

Please check Spark version on your cluster - it should be the same as version in pom.xml. Please also note, that all Spark dependencies should be marked as provided when you use spark-submit to deploy application

Upvotes: 7

JavaLearner
JavaLearner

Reputation: 215

As per the exception you are getting ,I think required jar is missing you need to add the required jar in your classpath which will resolve the issue.

refer this link to download the required jar

Upvotes: 1

Related Questions