CODEWITHSUNDEEP
apache-storm
mbbce
mbbce

Reputation: 2303

Apache Storm: Limit the number of tuples produced (in a certain time frame)

By my understanding ack, fail and nextTuple functions are called using the same thread in a tight loop as ISpout Javadoc suggest:

nextTuple, ack, and fail are all called in a tight loop in a single thread in the spout task.

Say we have a synthetic tuple generator and we want to limit the number of tuples emitted by the spout per second. How can that be achieved? Is putting a sleep() a good idea? Is there an alternative way?

Upvotes: 0

Views: 337

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

A sleep might not be the best idea because it blocks the spout to process incoming acks. See here: Why should I not loop or block in Spout.nextTuple()

I would just count the emitted tuples and remember a timestamp. If the number of tuples per time unit is exceeded and the time unit did not pass, just return from nextTuple() without emitting any tuple. When the time unit passed, reset the counter to zero, advance the timestamp by the time unit, and resume emitting. And so fourth.

Upvotes: 2

Related Questions