CODEWITHSUNDEEP
apache-storm
Hariprasad Taduru
Hariprasad Taduru

Reputation: 181

Storm - topology to topology

Is it possible or fine to emit the tuples from one topology to another topology?

Lets say in one topology, one specific bolt is doing storing of tuples into db. In another topology I don't want to duplicate or create the same bolt for storing the tuples. So from this second topology can I emit to first topology bolt?

-Hariprasad

Upvotes: 1

Views: 532

Answers (3)

aaryno
aaryno

Reputation: 636

The setup requires two storm topologies (A and B) and one Kafka topic. Let's call it "transfer"

Within the A topology where you want to send data to the B topology, use a Kafka producer:

[The kafka initialization code is taken directly from the docs: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example and obviously needs to be customized for your kafka installation.] 

public void Execute(Tuple input){
...
  Properties props = new Properties();
  props.put("metadata.broker.list", "broker1:9092,broker2:9092 ");
  props.put("serializer.class", "kafka.serializer.StringEncoder");
  props.put("partitioner.class", "example.producer.SimplePartitioner");
  props.put("request.required.acks", "1"); 

  ProducerConfig config = new ProducerConfig(props);

  Producer<String, String> producer = new Producer<String, String (config);
  String msg = ... 
  KeyedMessage<String, String> data = new KeyedMessage<String, String>
      ("transfers", ip, msg);
  producer.send(data);
  producer.close();

From Topology B, you create a Kafka Spout when you initialize your topology: 

BrokerHosts hosts = new ZkHosts(zkConnString);
SpoutConfig spoutConfig = new SpoutConfig(hosts, topicName, "/" + topicName, 
    UUID.randomUUID().toString());
spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

// Now it's just like any other spout
topologyBuilder.setSpout(kafkaSpout);

That requires running kafka, of course (check out https://kafka.apache.org/08/quickstart.html).

[Edit: Reading your question again: it sounds like you have a reusable component (save tuple) that you want to call from two different topologies and you are trying to call one from the other. Another approach is to offload this task to a third topology devoted to handling saving tuples and just create kafka messages of the items that need to be persisted within your topologies. In this way, ALL events to save-tuple will be handled the same way.]

Upvotes: 1

Ben Tse
Ben Tse

Reputation: 669

While you cannot directly pass tuples from one topology to another, you can use queuing system such as Apache Kafka to accomplish what you described. Storm has Kafka spout packaged in their latest releases.

Upvotes: 1

user2720864
user2720864

Reputation: 8171

This is currently not supported, you can not pass on tuples from one topology to another. Based on your use case why don't you use another bolt (within the same topology) subscribed to the db bolt instead of running a separate topology

Upvotes: 0

Related Questions