Running a script in apache storm

Question

Is there a way to simply run a python script in Apache Storm?

I'm trying to figure out how to use storm to run scripts but am having trouble. It seems like I need to create a Java program to call the script and use it as a bolt but I simply want to send a very basic python script to storm to see if it is possible.

I read that the following command is helpful in sending topologies to storm but am having trouble understanding the syntax and if I am allowed to send any python code to storm or if it needs to have specific syntax.

Can someone clarify whether or not I can submit any python script to storm and if so what the following line of code means.

storm shell resources/ python topology.py arg1 arg2

When I try to submit a basic python script using the above code i get the following output.

956  [main] INFO  backtype.storm.StormSubmitter - Uploading topology jar stormshell8691441.jar to assigned location: /home/scix3/apache/storm/data/nimbus/inbox/stormjar-ae0739f9-7c93-4f00-a02b-c4eceba3b005.jar
966  [main] INFO  backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /home/scix3/apache/storm/data/nimbus/inbox/stormjar-ae0739f9-7c93-4f00-a02b-c4eceba3b005.jar
Exception in thread "main" java.io.IOException: Cannot run program "simple.py" (in directory "."): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at java.lang.Runtime.exec(Runtime.java:617)
at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)
at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:254)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:319)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:386)
at backtype.storm.command.shell_submission$_main.doInvoke(shell_submission.clj:29)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at backtype.storm.command.shell_submission.main(Unknown Source)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 10 more

The exact command I'm using (possibly incorrect) is storm shell resources/ simple.py

simple.py is merely a print 'Hello, world script.

I'm using storm version 0.9.4

Answer 1

try pyleus (https://github.com/Yelp/pyleus) or streamparse (https://github.com/Parsely/streamparse), i will recommend using pyleus as it is simple.

Answer 2

Yes, you can run python on Storm. In fact you can run just about code from just about any language on a storm cluster, its just a matter of implementing the API.

However, there are some requirements for that to work, and so far as I can tell said requirements are not spelled out in the storm documentation. The fastest path to get up an running would be to take the splitsentence.py example from the storm source and run with it.