CODEWITHSUNDEEP
apache-stormstormcrawlerapache-storm-topology
aeranginkaman
aeranginkaman

Reputation: 289

how to use python bolt in storm crawler?

I have some image classifiers that have been written in python. A lot of examples are available on the web which describes the way of using python in storm bolt that uses from stdin/stdout. I want to integrate my python image classifier with storm crawler topology. Is it possible or not?

Thanks

Upvotes: 0

Views: 404

Answers (2)

Ali AzG
Ali AzG

Reputation: 1983

Yes, you can. if you are using Flux, this is a sample definition of how to use a python bolt in your topology:

  - id: "pythonbolt"
    className: "org.apache.storm.flux.wrappers.bolts.FluxShellBolt"
    constructorArgs:
      - ["python", "/absolute/path/to/your/python_file.py"]
      # declare your outputs here:
      - ["output0", "output1", "output2"]
    parallelism: 1

NOTE: Make sure you emit simple data types (like string, integer, etc.) to your python bolt. not a java data type, or it'll throw errors! first download storm.py form

here

And this is a sample python bolt:

import storm

class SampleBolt(storm.BasicBolt):
    # Initialize this instance
    def initialize(self, conf, context):
        self._conf = conf
        self._context = context

    def process(self, tup):
        # Some processes here, and then emit your outputs.
        storm.emit([output0, output1, output2])

# Start the bolt when it's invoked
SampleBolt().run()

Upvotes: 1

Julien Nioche
Julien Nioche

Reputation: 4864

Definitely possible, did that a few years ago to integrate an image classifier with Tensorflow into a StormCrawler topology. Can't remember the details and the code stayed with the customers I wrote it for but it was based on the multilang protocol, don't remember the details unfortunately.

Upvotes: 1

Related Questions