Storm is crashing due to zookeeper status changing

Question

I have very basic storm topology which contains 1 unreliable spout + 1 bolt. Spout is fed by Kafka broker via zookeeper. The Spout sends tuples to a bolt witch in its role do nothing ( has empty execute method). The topology run in local cluster mode and it crashes after a while from run. I tried to increase the app memory by -Xmx parameter many times, but the problem still appears. please help! Here is the code snippet that build the topology:

builder.setSpout("kafka_forwarding_spout", new KafkaForwardingSpout(),1);
      builder.setBolt("fake_bolt", new FakeBolt(),1).localOrShuffleGrouping("kafka_forwarding_spout");

Here is a snippet from storm log:

1318482 [Thread-37-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
1319574 [CuratorFramework-9] INFO  com.netflix.curator.framework.state.ConnectionStateManager - State change: LOST
1312307 [Thread-39-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
1307441 [Thread-25-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
1227216 [Thread-29-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
1191153 [Thread-31-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
1771836 [Thread-39-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
1688630 [ConnectionStateManager-0] WARN  com.netflix.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
1650422 [Thread-35-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
1635500 [Thread-19-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
1616369 [CuratorFramework-11] ERROR com.netflix.curator.framework.imps.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
  at com.netflix.curator.ConnectionState.getZooKeeper(ConnectionState.java:72) ~[curator-client-1.0.1.jar:na]
  at com.netflix.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:74) ~[curator-client-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:353) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.BackgroundSyncImpl.performBackgroundOperation(BackgroundSyncImpl.java:39) ~[curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:40) ~[curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:547) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.CuratorFrameworkImpl.access$200(CuratorFrameworkImpl.java:50) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.CuratorFrameworkImpl$2.call(CuratorFrameworkImpl.java:177) [curator-framework-1.0.1.jar:na]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_05]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_05]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_05]
  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
1617025 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] WARN  org.apache.zookeeper.server.NIOServerCnxn - EndOfStreamException: Unable to read additional data from client sessionid 0x149dd7eb6a2000b, likely client has closed socket
1600212 [Thread-23-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
1584086 [Thread-21-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
1570224 [Thread-9] INFO  backtype.storm.daemon.worker - Shutting down receive thread
1569786 [ConnectionStateManager-0] WARN  com.netflix.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
2158302 [Thread-5] ERROR backtype.storm.event - Error when processing event
java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
  at backtype.storm.util$wrap_in_runtime.invoke(util.clj:44) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
  at backtype.storm.zookeeper$get_children.invoke(zookeeper.clj:133) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
  at backtype.storm.cluster$mk_distributed_cluster_state$reify__2385.get_children(cluster.clj:101) ~[na:na]
  at backtype.storm.cluster$mk_storm_cluster_state$reify__2814.assignments(cluster.clj:243) ~[na:na]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_05]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_05]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_05]
  at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_05]
  at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.4.0.jar:na]
  at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.4.0.jar:na]
  at backtype.storm.daemon.supervisor$assignments_snapshot.invoke(supervisor.clj:39) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
  at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__6251.invoke(supervisor.clj:298) ~[na:na]
  at backtype.storm.event$event_manager$fn__2921.invoke(event.clj:39) ~[na:na]
  at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
  at com.netflix.curator.ConnectionState.getZooKeeper(ConnectionState.java:72) ~[curator-client-1.0.1.jar:na]
  at com.netflix.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:74) ~[curator-client-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:353) ~[curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:180) ~[curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:173) ~[curator-framework-1.0.1.jar:na]
  at com.netflix.curator.RetryLoop.callWithRetry(RetryLoop.java:85) ~[curator-client-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:169) ~[curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:161) ~[curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:36) ~[curator-framework-1.0.1.jar:na]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_05]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_05]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_05]
  at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_05]
  at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.4.0.jar:na]
  at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.4.0.jar:na]
  at backtype.storm.zookeeper$get_children.invoke(zookeeper.clj:131) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
  ... 13 common frames omitted
2117747 [Thread-4] ERROR com.netflix.curator.ConnectionState - Connection timed out
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
  at com.netflix.curator.ConnectionState.getZooKeeper(ConnectionState.java:72) ~[curator-client-1.0.1.jar:na]
  at com.netflix.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:74) [curator-client-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:353) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:149) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:138) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.RetryLoop.callWithRetry(RetryLoop.java:85) [curator-client-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:134) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:125) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:34) [curator-framework-1.0.1.jar:na]
  at backtype.storm.zookeeper$exists_node_QMARK_$fn__2170.invoke(zookeeper.clj:96) [na:0.9.1-incubating]
  at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:93) [na:0.9.1-incubating]
  at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:107) [na:0.9.1-incubating]
  at backtype.storm.cluster$mk_distributed_cluster_state$reify__2385.set_ephemeral_node(cluster.clj:69) [na:0.9.1-incubating]
  at backtype.storm.cluster$mk_storm_cluster_state$reify__2814.supervisor_heartbeat_BANG_(cluster.clj:315) [na:0.9.1-incubating]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_05]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_05]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_05]
  at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_05]
  at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) [clojure-1.4.0.jar:na]
  at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) [clojure-1.4.0.jar:na]
  at backtype.storm.daemon.supervisor$eval6298$exec_fn__1459__auto____6299$heartbeat_fn__6301.invoke(supervisor.clj:378) [na:0.9.1-incubating]
  at backtype.storm.timer$schedule_recurring$this__2141.invoke(timer.clj:92) [na:0.9.1-incubating]
  at backtype.storm.timer$mk_timer$fn__2124$fn__2125.invoke(timer.clj:48) [na:0.9.1-incubating]
  at backtype.storm.timer$mk_timer$fn__2124.invoke(timer.clj:41) [na:0.9.1-incubating]
  at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
2078727 [Thread-16] ERROR com.netflix.curator.ConnectionState - Connection timed out
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
  at com.netflix.curator.ConnectionState.getZooKeeper(ConnectionState.java:72) ~[curator-client-1.0.1.jar:na]
  at com.netflix.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:74) [curator-client-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:353) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:149) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:138) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.RetryLoop.callWithRetry(RetryLoop.java:85) [curator-client-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:134) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:125) [curator-framework-1.0.1.jar:na]
  at com.netflix.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:34) [curator-framework-1.0.1.jar:na]
  at backtype.storm.zookeeper$exists_node_QMARK_$fn__2170.invoke(zookeeper.clj:96) [na:0.9.1-incubating]
  at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:93) [na:0.9.1-incubating]
  at backtype.storm.zookeeper$exists.invoke(zookeeper.clj:141) [na:0.9.1-incubating]
  at backtype.storm.cluster$mk_distributed_cluster_state$reify__2385.set_data(cluster.clj:85) [na:0.9.1-incubating]
  at backtype.storm.cluster$mk_storm_cluster_state$reify__2814.worker_heartbeat_BANG_(cluster.clj:291) [na:0.9.1-incubating]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_05]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_05]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_05]
  at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_05]
  at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) [clojure-1.4.0.jar:na]
  at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) [clojure-1.4.0.jar:na]
  at backtype.storm.daemon.worker$do_executor_heartbeats.doInvoke(worker.clj:53) [na:0.9.1-incubating]
  at clojure.lang.RestFn.invoke(RestFn.java:439) [clojure-1.4.0.jar:na]
  at backtype.storm.daemon.worker$eval5665$exec_fn__1459__auto____5666$fn__5669.invoke(worker.clj:367) [na:0.9.1-incubating]
  at backtype.storm.timer$schedule_recurring$this__2141.invoke(timer.clj:92) [na:0.9.1-incubating]
  at backtype.storm.timer$mk_timer$fn__2124$fn__2125.invoke(timer.clj:48) [na:0.9.1-incubating]
  at backtype.storm.timer$mk_timer$fn__2124.invoke(timer.clj:41) [na:0.9.1-incubating]
  at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
2079838 [CuratorFramework-11] INFO  com.netflix.curator.framework.state.ConnectionStateManager - State change: LOST
2052508 [Thread-37-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
1948998 [yassin-long-d_localhost.localdomain-1416760311632-aa3d627e_watcher_executor] INFO  kafka.consumer.ZookeeperConsumerConnector - [yassin-long-d_localhost.localdomain-1416760311632-aa3d627e], exception during rebalance 
java.lang.NullPointerException: null
  at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103) ~[zkclient-0.3.jar:0.3]
  at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770) ~[zkclient-0.3.jar:0.3]
  at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766) ~[zkclient-0.3.jar:0.3]
  at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[zkclient-0.3.jar:0.3]
  at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766) ~[zkclient-0.3.jar:0.3]
  at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761) ~[zkclient-0.3.jar:0.3]
  at kafka.utils.ZkUtils$.readData(ZkUtils.scala:461) ~[kafka_2.10-0.8.1.jar:na]
  at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:505) ~[kafka_2.10-0.8.1.jar:na]
  at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504) ~[kafka_2.10-0.8.1.jar:na]
  at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[scala-library-2.10.1.jar:na]
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[scala-library-2.10.1.jar:na]
  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[scala-library-2.10.1.jar:na]
  at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[scala-library-2.10.1.jar:na]
  at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504) ~[kafka_2.10-0.8.1.jar:na]
  at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407) ~[kafka_2.10-0.8.1.jar:na]
  at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) [scala-library-2.10.1.jar:na]
  at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402) [kafka_2.10-0.8.1.jar:na]
  at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:355) [kafka_2.10-0.8.1.jar:na]
1872056 [Thread-39-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
1853312 [Thread-35-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
1840380 [Thread-31-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
1808599 [Thread-41-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
1799622 [Thread-25-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
2287292 [Thread-41-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
2287075 [Thread-37-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
2286862 [yassin-long-d_localhost.localdomain-1416760311632-aa3d627e_watcher_executor] INFO  kafka.consumer.ZookeeperConsumerConnector - [yassin-long-d_localhost.localdomain-1416760311632-aa3d627e], end rebalancing consumer yassin-long-d_localhost.localdomain-1416760311632-aa3d627e try #0
2280400 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] WARN  org.apache.zookeeper.server.NIOServerCnxn - EndOfStreamException: Unable to read additional data from client sessionid 0x149dd7eb6a20005, likely client has closed socket
2280180 [ConnectionStateManager-0] WARN  com.netflix.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
2267345 [Thread-9] INFO  backtype.storm.messaging.loader - Shutting down receiving-thread: [events_topology-1-1416760309, 1027]
2242225 [Thread-19-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2224569 [Thread-29-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2226559 [Thread-21-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
2209276 [Thread-23-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2296607 [Thread-41-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2293718 [Thread-37-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
2292830 [Thread-25-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2498572 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] WARN  org.apache.zookeeper.server.NIOServerCnxn - EndOfStreamException: Unable to read additional data from client sessionid 0x149dd7eb6a20003, likely client has closed socket
2540517 [Thread-41-kafka_forwarding_spout-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Error while calling watcher 
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main-SendThread(localhost:2001)"
2637439 [Thread-35-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
2647739 [Thread-39-kafka_forwarding_spout-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (Disconnected)
2672605 [main-SendThread(localhost:2001)] ERROR org.apache.zookeeper.ClientCnxn - from main-SendThread(localhost:2001)
java.lang.OutOfMemoryError: Java heap space
2764243 [Thread-25-kafka_forwarding_spout-SendThread(zoo2:2182)] ERROR org.apache.zookeeper.ClientCnxn - from Thread-25-kafka_forwarding_spout-SendThread(zoo2:2182)
java.lang.OutOfMemoryError: Java heap space
2720032 [main-SendThread(localhost:2001)] ERROR org.apache.zookeeper.ClientCnxn - from main-SendThread(localhost:2001)
java.lang.OutOfMemoryError: Java heap space
2731157 [Thread-8] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[Thread-8,5,main] died
java.lang.OutOfMemoryError: Java heap space
2800693 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001,5,main] died

Answer 1

This issue is being caused due to the Java Virtual Machine running out of available Heap memory in the system. Depending on on which underlying operating system that you are using, you can use the -Xmx parameter followed by the amount of memory you want to allocate. For example, if you want to allocate 2GB of memory then you need to mention -Xms2048m and the maximum heap size to be set as -Xmx2048m. However, remember that increasing the heap size after getting an OutofMemoryError is generally a short-term solution and it is highly probable to occur again in the future. It is generally an indicator for optimising your code to be memory efficient.

Answer 2

Seems like a java heap size problem. Check there are many other configurable memory related parameters.