CODEWITHSUNDEEP
pythonmatplotlibapache-stormtopology
Erol
Erol

Reputation: 6526

"No module named" error in Apache Storm Python Bolt

I am getting the following error message when I try to locally run a Storm topology with a single bolt implemented in python. I am doing everything like in the example of WordCountTopology in storm-starter kit but it fails to load the modules like matplotlib that are required in my python bolt. Any help or guidance would be appreciated.

I am using Anaconda on Windows machine, if that helps.

12970 [Thread-21-divide] ERROR backtype.storm.daemon.executor - 
java.lang.RuntimeException: Error when launching multilang subprocess
Traceback (most recent call last):
  File "pythonBolt.py", line 4, in <module>
    from matplotlib import mlab
ImportError: No module named matplotlib

    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:66) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.daemon.executor$fn__8077$fn__8090.invoke(executor.clj:746) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.util$async_loop$fn__543.invoke(util.clj:473) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_31]
Caused by: java.io.IOException: Die Pipe wurde beendet
    at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_31]
    at java.io.FileOutputStream.write(Unknown Source) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flushBuffer(Unknown Source) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flush(Unknown Source) ~[na:1.8.0_31]
    at java.io.DataOutputStream.flush(Unknown Source) ~[na:1.8.0_31]
    at backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    ... 5 common frames omitted

Upvotes: 3

Views: 874

Answers (4)

chans.best
chans.best

Reputation: 13

its 2018 and i was stuck for hours for same issue , you need to ensure two things 1) you need to install matplot lib in any of python setups on your machine. so this setup can be done either in default /usr/bin/python or if you have anaconda installation you may do it in /home/user/anaconda3/bin/python or if you have a virtualenv setup then you might install it there. 2)When writing a shell bolt implementation in java code make sure to provide path to python installation where you have installed matplotlib. for eg.below would be code if you installed library in anaconda at /home/user/anaconda3/bin/python

public static class SplitSentence extends ShellBolt implements IRichBolt {

    public SplitSentence() {
      super("home/user/anaconda3/bin/python", "splitsentence.py");
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
      declarer.declare(new Fields("word"));
    }

    @Override
    public Map<String, Object> getComponentConfiguration() {
      return null;
    }
  }

credits and sources :

In storm, how to specify specific version of python

http://storm.apache.org/releases/2.0.0-SNAPSHOT/Multilang-protocol.html

Upvotes: 1

Shaobo
Shaobo

Reputation: 1

Storm use the system default python instead of anaconda python. You installed matplotlib in anaconda python. You did not install matplotlib in system python, so when storm use system python it will show "no module named" error.

Upvotes: 0

red_devil
red_devil

Reputation: 1049

Well, I was hit with same issue . I got around by removing anaconda from my supervisor nodes and installing the required packages (i.e - matplotlib,numpy,pandas etc) manually. It's certainly not the best way of doing things, but it works.

Upvotes: 0

lapinkoira
lapinkoira

Reputation: 8998

Are you using python virtual enviorments? In any case you need to install matplotlib, you can do that with running pip install matplotlib

Upvotes: 1

Related Questions