How to debug Apache Storm in Local Cluster/Mode through eclipse

Question

Using the following Q&A I managed to get debugging enabled through eclipse on an Apache Storm cluster (running locally). How to debug Apache Storm in Eclipse?

My conf/storm.yaml has the following line to enable debugging on the worker nodes:

worker.childopts: "-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=y"

When I submit a Topology to storm to run (in a cluster), I can set breakpoints and view variables in my editor.

But when I try to run it locally (In Local Mode), I can't seem to connect (Connection Refused) - through eclipse.

# I'm using storm crawler, I submit a topology like so:
storm jar target/storm-crawler-core-10.6-SNAPSHOT-jar-with-dependencies.jar \
 com.digitalpebble.storm.crawler.CrawlTopology \
-conf crawler-conf.yaml \
-local

# ^ The `-local` runs it in a `LocalCluster`
# If I submit it to my actual cluster (without -local), I can debug it through eclipse.

# View the pastebin for all the output : http://pastebin.com/PEdA7fH0

I have included all the output from the above command to a pastebin. Click here to view it

More information on how storm crawler launches the LocalCluster.

I want to be able to debug in local mode so I can see the output in the command line (as I make my way through the breakpoints I set) and quickly make changes and re-run, basically to speed up my development flow.

---

How can I debug my code through the eclipse debugger when running Apache Storm in a LocalCluster (Local Mode)?

Answer 1

In local mode, you can use e.g.

export STORM_JAR_JVM_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=localhost:8000"

before calling the storm jar command to debug remotely with Eclipse.

Answer 2

We can debug Storm Topology like any other java code in Eclipse. I launch my topology with in Eclipse itself using LocalCluster mode. I'm able to debug ShellBolts, Trident bolts etc.

In addition I also have storm-core added as project in my Eclipse (instead of maven dependency), which enables me to debug and understand whats happening inside storm. Useful to better track Tuple acknowledgements, emit handlers, anchors, timeouts, exceptions etc.

Answer 3

If you run in local mode, there is no worker JVM involved, ie, no worker process is stared. Thus, your worker.childopts settings do not have any effect.

The easiest way to debug within Eclipse it to submit/start the topology within Eclipse instead of command line. Your class CrawlTopology has a main method, thus you can just execute it directly in Eclipse (of course in debug mode). You don't need to specify a jar file name. Just specify your options -conf crawler-conf.yaml -local in your Eclipse run configuration.