CODEWITHSUNDEEP
apache-storm
Harsh Kumar
Harsh Kumar

Reputation: 138

Storm bolt following a kafka bolt

I have a Storm topology where I have to send output to kafka as well as update a value in redis. For this I have a Kafkabolt as well as a RedisBolt. Below is what my topology looks like - 

tp.setSpout("kafkaSpout", kafkaSpout, 3);

tp.setBolt("EvaluatorBolt", evaluatorBolt, 6).shuffleGrouping("kafkaStream");

tp.setBolt("ResultToRedisBolt",ResultsToRedisBolt,3).shuffleGrouping("EvaluatorBolt","ResultStream");

tp.setBolt("ResultToKafkaBolt", ResultsToKafkaBolt, 3).shuffleGrouping("EvaluatorBolt","ResultStream");

The problem is that both of the end bolts (Redis and Kafka) are listening to the same stream from the preceding bolt (ResultStream), hence both can fail independently. What I really need is that if the result is successfully published in Kafka, then only I update the value in Redis. Is there a way to have an output stream from a kafkaBolt where I can get the messages published successfully to Kafka? I can then probably listen to that stream in my RedisBolt and act accordingly.

Upvotes: 0

Views: 166

Answers (1)

Stig Rohde Døssing
Stig Rohde Døssing

Reputation: 3651

It is not currently possible, unless you modify the bolt code. You would likely be better off changing your design slightly, since doing extra processing after the tuple is written to Kafka has some drawbacks. If you write the tuple to Kafka and you fail to write to Redis, you will get duplicates in Kafka, since the processing will start over at the spout.

It might be better, depending on your use case, to write the result to Kafka, and then have another topology read the result from Kafka and write to Redis.

If you still need to be able to emit new tuples from the bolt, it should be pretty easy to implement. The bolt recently got the ability to add a custom Producer callback, so we could extend that mechanism.

See the discussion at https://github.com/apache/storm/pull/2790#issuecomment-411709331 for context.

Upvotes: 1

Related Questions