Reputation: 13
Apache storm - java.lang.NoClassDefFoundError: com/google/gson/Gson
I'm using Apache Storm 0.9.4. I set up a 5 nodes cluster and it works fine. (Actually 5 supervisors are working as docker containers on 5 different physical nodes.)
My environment is here
$cat /etc/redhat-release
CentOS release 6.6 (Final)
$docker -v
Docker version 1.4.1, build 5bc2ff8/1.4.1
$java -version
java version "1.7.0_79" OpenJDK Runtime Environment
(rhel-2.5.5.1.el6_6-x86_64 u79-b14) OpenJDK 64-Bit Server VM (build
24.79-b02, mixed mode)
Next, I create a maven project to submit a topology using Netbeans.
There are 1 topology and 1 spout.(very simple)
- MyTopology.java
- LoggerSpout.java
MyTopology works fine without GSON.
But, after I added Gson gson = new Gson(); in the nextTuple() function,
I got java.lang.NoClassDefFoundError: com/google/gson/Gson.
Here is what I did
-
1. mvn clean -> mvn compile -> mvn package -> BUILD SUCCESSFULY
2. storm jar ~/Desktop/teststorm/target/teststorm-1.0.jar jp.soushi.teststorm.MyTopology -> Submitting is OK.
Running: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/bin/java -client -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-0.9.4 -Dstorm.log.dir=/usr/local/apache-storm-0.9.4/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-0.9.4/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.4/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.4/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.4/lib/clj-stacktrace-0.2.2.jar:/usr/local/apache-storm-0.9.4/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.4/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.4/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.4/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.4/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.4/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.4/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.4/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.4/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.4/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.4/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.4/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.4/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.4/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.4/lib/json-simple-1.1.jar:/usr/local/apache-storm-0.9.4/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.4/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.4/lib/logback-classic-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/logback-core-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.4/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.4/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.4/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.4/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.4/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.4/lib/slf4j-api-1.7.5.jar:/usr/local/apache-storm-0.9.4/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.4/lib/storm-core-0.9.4.jar:/usr/local/apache-storm-0.9.4/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.4/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.4/lib/tools.macro-0.1.0.jar:/Users/soushi/Desktop/teststorm/target/teststorm-1.0.jar:/Users/soushi/.storm:/usr/local/apache-storm-0.9.4/bin -Dstorm.jar=/Users/soushi/Desktop/teststorm/target/teststorm-1.0.jar jp.soushi.teststorm.MyTopology
491 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
499 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar /Users/soushi/Desktop/teststorm/target/teststorm-1.0.jar to assigned location: storm-local/nimbus/inbox/stormjar-b6added0-ffb8-4602-9d89-b567ed87d335.jar
509 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: storm-local/nimbus/inbox/stormjar-b6added0-ffb8-4602-9d89-b567ed87d335.jar
509 [main] INFO backtype.storm.StormSubmitter - Submitting topology MyTopology in distributed mode with conf {"topology.workers":31,"topology.debug":true,"topology.max.spout.pending":5000}
766 [main] INFO backtype.storm.StormSubmitter - Finished submitting topology: MyTopology
- I checked the topology using Storm Web UI. After a while, I got NoClassDefFoundError.
- I edited nimbus.yaml, and add
java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib/java". and make sure /usr/lib/java can be accessible in the docker container.
Here is the error log(worker-6727.log).
2015-04-28T10:05:18.497+0000 b.s.d.executor [ERROR]
java.lang.NoClassDefFoundError: com/google/gson/Gson
at jp.soushi.adcontex.SensorSpout.nextTuple(SensorSpout.java:43) ~[stormjar.jar:na]
at backtype.storm.daemon.executor$fn__4654$fn__4669$fn__4698.invoke(executor.clj:565) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.ClassNotFoundException: com.google.gson.Gson
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_79]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_79]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_79]
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_79]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_79]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_79]
... 5 common frames omitted
2015-04-28T10:05:18.564+0000 b.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__5102$fn__5103.invoke(worker.clj:495) [storm-core-0.9.4.jar:0.9.4]
at backtype.storm.daemon.executor$mk_executor_data$fn__4555$fn__4556.invoke(executor.clj:240) [storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:473) [storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
- pom.xml I tried changing gson's scope(runtime, provided, compile), but it still doesn't work
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>jp.soushi</groupId>
<artifactId>teststorm</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<name>teststorm</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>jp.soushi.adcontex.AdContexTopology</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy</id>
<phase>install</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>central</id>
<url>https://repo1.maven.org/maven2</url>
<releases>
<enabled>true</enabled>
</releases>
</repository>
<repository>
<id>github-releases</id>
<url>http://oss.sonatype.org/content/repositories/github-releases/</url>
</repository>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
<repository>
<id>local-project-libraries</id>
<name>Local project libraries</name>
<url>file://${project.basedir}/lib</url>
<layout>default</layout>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>storm</groupId>
<artifactId>storm-lib</artifactId>
<version>0.9.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3.1</version>
<type>jar</type>
<scope>runtime</scope>
</dependency>
</dependencies>
</project>
- MyTopology.java
package jp.soushi.teststorm;
import backtype.storm.Config;
import backtype.storm.StormSubmitter;
import backtype.storm.topology.TopologyBuilder;
public class MyTopology {
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("Logs", new LoggerSpout(), 5);
Config conf = new Config();
conf.setDebug(true);
conf.setNumWorkers(31);
conf.setMaxSpoutPending(5000);
StormSubmitter.submitTopology( "MyTopology", conf, builder.createTopology() );
}
}
- LoggerSpout.java
package jp.soushi.teststorm;
import backtype.storm.Config;
import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseRichSpout;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;
import com.google.gson.Gson;
import java.util.HashMap;
import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerSpout extends BaseRichSpout {
public static Logger LOG = LoggerFactory.getLogger(LoggerSpout.class);
boolean _isDistributed;
SpoutOutputCollector _collector;
public LoggerSpout() {
this(true);
}
public LoggerSpout(boolean isDistributed) {
_isDistributed = isDistributed;
}
public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
_collector = collector;
}
public void close() {
}
public void nextTuple() {
Utils.sleep(100);
Gson gson = new Gson();
_collector.emit(new Values("Test"));
}
public void ack(Object msgId) {
}
public void fail(Object msgId) {
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("Logs"));
}
@Override
public Map<String, Object> getComponentConfiguration() {
if(!_isDistributed) {
Map<String, Object> ret = new HashMap<String, Object>();
ret.put(Config.TOPOLOGY_MAX_TASK_PARALLELISM, 1);
return ret;
} else {
return null;
}
}
}
I found this answer. But I don't understand well. java.lang.NoClassDefFoundError: com/google/gson/Gson
I also read these answers.
- Getting java.lang.ClassNotFoundException: com.google.gson.Gson for maven project
- GSON is not being imported into the maven pproject
- http://www.ciiycode.com/0JSJ6PgUXWxq/javalangnoclassdeffounderror-comgooglegsongsonbuilder-error
I spend 3 days to solve that... I have no idea. Can anyone provide any solution to this.Thanks in advance.
Upvotes: 1
Views: 2348
Answers (3)
Reputation: 10618
The reason of the NoClassDefFoundError is that the gson dependency is not included in your jar file and you need to send your topology to your Storm cluster with all the required dependencies in it.
What you want to do is to create a "fat jar". To do so, you are going to need the Maven Shade plugin. However, you should be careful not to include the storm-core dependency in your package, which would conflict with Storm itself.
So, you should add this to your pom.xml:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.storm:storm-core</exclude>
</excludes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Running mvn package should now produce a jar file that includes all dependencies (except storm-core). You can verify this with:
jar -tf ~/Desktop/teststorm/target/teststorm-1.0.jar
Upvotes: 0
Reputation: 1666
You have to run topology with fat jar including all the dependency classes which your topology uses. Otherwise, your topology will not be able to find them in runtime.
You've already tried to build fat jar but it's very likely that your jar did not include Gson. You can check with jar tf YOUR.jar command.
For that purpose, I've been using maven-shade-plugin. Refer to : https://maven.apache.org/plugins/maven-shade-plugin/
Upvotes: 2
Reputation: 2110
All cases show one common thing problem is with <scope>.
In your case you have provided scope as 'runtime'. This means that gson library will not be required while compiling but in runtime, which was not desired .
In your case you need to change your to compile. ie
<scope>compile</scope>
Read more below :
There are 6 scopes available:
compile
This is the default scope, used if none is specified. Compile dependencies are available in all classpaths of a project. Furthermore, those dependencies are propagated to dependent projects.
provided
This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.
runtime
This scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath.
test
This scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases.
system
This scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository.
import (only available in Maven 2.0.9 or later)
This scope is only used on a dependency of type pom in the section. It indicates that the specified POM should be replaced with the dependencies in that POM's section. Since they are replaced, dependencies with a scope of import do not actually participate in limiting the transitivity of a dependency.
Upvotes: 0
Related Questions
- Get NoClassDefFound Error in Apache Maven 3.8.6 (JDK-19+Gson2.10)
- Storm ClassNotFoundException
- java.lang.NoClassDefFoundError caused by Gson
- java.lang.NoClassDefFoundError: org/yaml
- Handler processing failed; nested exception is java.lang.NoClassDefFoundError: com/google/gson/Gson
- java.lang.NoClassDefFoundError: com/sun/tools/corba/se/idl/InvalidArgument
- java.lang.NoClassDefFoundError
- Storm cannot find class
- Getting java.lang.NoClassDefFoundError while using Google Gson library
- java.lang.NoClassDefFoundError: com/google/gson/Gson