java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging

Question

My Spark Streaming program received the following error: Exception in thread "main" java.lang.NoClassDefFoundError:org/apache/spark/internal/Logging My version of spark is 2.1, which is the same as the version running in the cluster.

The information I found on the Internet prompted me that the old version of org.apache.spark.Logging became org.apache.spark.internal.Logging in the new version, which prevented the jar package from being found. But the dependency introduced in my pom is a new version. Why can't I find the jar package?

    <properties>
            <spark.version>2.1.0</spark.version>
            <scala.version>2.11</scala.version>
    </properties>

    <dependencies>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-core_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-streaming_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-sql_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-hive_${scala.version}</artifactId>
                    <version>${spark.version}</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
                    <version>2.1.0</version>
            </dependency>
            <dependency>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-client</artifactId>
                    <version>2.6.0</version>
            </dependency>
            <dependency>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.15.2</version>
            </dependency>
            <dependency>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                    <version>1.2.17</version>
            </dependency>
    </dependencies>
    <build>
            <plugins>
                    <plugin>
                            <groupId>org.scala-tools</groupId>
                            <artifactId>maven-scala-plugin</artifactId>
                            <version>2.15.2</version>
                            <executions>
                                    <execution>
                                            <goals>
                                                    <goal>compile</goal>
                                                    <goal>testCompile</goal>
                                            </goals>
                                    </execution>
                            </executions>
                    </plugin>

                    <plugin>
                            <artifactId>maven-compiler-plugin</artifactId>
                            <version>3.6.0</version>
                            <configuration>
                                    <source>1.8</source>
                                    <target>1.8</target>
                            </configuration>
                    </plugin>
                    <plugin>
                            <groupId>org.apache.maven.plugins</groupId>
                            <artifactId>maven-surefire-plugin</artifactId>
                            <version>2.19</version>
                            <configuration>
                                    <skip>true</skip>
                            </configuration>
                    </plugin>
            </plugins>
    </build>

Answer 1

I have encountered java.lang.NoClassDefFoundError several times.I have Spark-2.3.1 installed so I think it should work in your case too.

In my case, java.lang.NoClassDefFoundError:org/apache/spark/internal/Logging should come from spark-core_2.11-2.3.1.jar and in your case, it should be coming from spark-core_2.11-2.1.0.jar based on your spark and scala version mentioned in the question

When you look at the source code, it is using org.slf4j._ classes. So my recommendation would be to add this dependency in your pom and then try. For NoClassDefFoundError, it is always better to find the jar which is generating this error and then try to backtrack it.

Below is how you can find out which jar is causing the NoClassDefFound error, assuming you have all the dependency jars in ~/spark/jars location.

for i in `ls ~/spark/jars/*.jar`; 
  do 
  jar tvf $i | grep  org/apache/spark/internal/Logging; 
  echo "************** $i ************** "; 
done

Please let me know if this solved your issue.