Storm upgrade - issue with zookeeper

Question

After upgrading from storm 0.9.1 to 0.9.2 I got the following exception while trying to run nimbus:

2014-06-29 16:31:47 b.s.d.nimbus [ERROR] Error when processing event
java.lang.RuntimeException: java.io.InvalidClassException: backtype.storm.daemon.common.SupervisorInfo; local class incompatible: stream classdesc serialVersionUID = 7648414326720210054, local class serialVersionUID = 7463898661547835557
        at backtype.storm.utils.Utils.deserialize(Utils.java:93) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.cluster$maybe_deserialize.invoke(cluster.clj:200) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.cluster$mk_storm_cluster_state$reify__2284.supervisor_info(cluster.clj:299) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_60]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_60]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_60]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_60]
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.5.1.jar:na]
        at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.nimbus$all_supervisor_info$fn__4715.invoke(nimbus.clj:277) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at clojure.core$map$fn__4207.invoke(core.clj:2487) ~[clojure-1.5.1.jar:na]
        at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
        at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
        at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.5.1.jar:na]
        at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.5.1.jar:na]
        at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
        at clojure.core$mapcat.doInvoke(core.clj:2514) ~[clojure-1.5.1.jar:na]
        at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.nimbus$all_supervisor_info.invoke(nimbus.clj:275) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.nimbus$all_scheduling_slots.invoke(nimbus.clj:288) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.nimbus$compute_new_topology__GT_executor__GT_node_PLUS_port.invoke(nimbus.clj:580) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:660) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.nimbus$fn__5210$exec_fn__1396__auto____5211$fn__5216$fn__5217.invoke(nimbus.clj:905) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.nimbus$fn__5210$exec_fn__1396__auto____5211$fn__5216.invoke(nimbus.clj:904) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.timer$schedule_recurring$this__1134.invoke(timer.clj:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.timer$mk_timer$fn__1117$fn__1118.invoke(timer.clj:50) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.timer$mk_timer$fn__1117.invoke(timer.clj:42) [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
Caused by: java.io.InvalidClassException: backtype.storm.daemon.common.SupervisorInfo; local class incompatible: stream classdesc serialVersionUID = 7648414326720210054, local class serialVersionUID = 7463898661547835557
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617) ~[na:1.7.0_60]
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622) ~[na:1.7.0_60]
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) ~[na:1.7.0_60]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) ~[na:1.7.0_60]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) ~[na:1.7.0_60]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) ~[na:1.7.0_60]
        at backtype.storm.utils.Utils.deserialize(Utils.java:89) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
        ... 29 common frames omitted
2014-06-29 16:31:47 b.s.util [INFO] Halting process: ("Error when processing an event")

After killing QuorumPeerMain nimbus was able to run. Looks like something is wrong with the zookeeper, or some traces from old zk versions are still cached somewhere.

What could cause this issue?

Answer 1

Finally found the reason why.

Quick solution : remove storm node recursively from your zookeeper and rerun.

Old data in zookeeper storm node conflicts with newly installed storm, and that causes serialVersionUID mismatch.

referred to https://github.com/Parsely/streamparse/issues/27