CODEWITHSUNDEEP
apache-stormapache-kafka
Vivek Rao
Vivek Rao

Reputation: 586

Turn off acking in KafkaSpout while respecting maxSpoutPending

I'm using storm 0.9.3. I'm trying to turn off acking per tuple for my topology. I set Config.TOPOLOGY_ACKER_EXECUTORS to 0, and maxSpoutPending to 500. When I run my topology, I'm noticing that maxSpoutPending is being ignored and the spout continues emitting well past that limit. Here's my config -

config.setNumWorkers(3);
config.setMaxSpoutPending(500);
config.put("topology.sleep.spout.wait.strategy.time.ms", 50);
config.put("topology.message.timeout.secs", 300);
config.put(Config.TOPOLOGY_ACKER_EXECUTORS, 0);

I'm using KafkaSpout to read from Kafka and a single bolt to consume the message.

Upvotes: 3

Views: 2359

Answers (1)

Shams
Shams

Reputation: 3677

By setting TOPOLOGY_ACKER_EXECUTORS to 0, storm will acknowledge all tuples immediately when they come off the spout, which may not be reliable, because no mechanism will work to check if tuple is processed or failed.
And by setting setMaxSpoutPending tells storm the maximum number of tuples pending on spout to process. MaxSpoutPending dosen't limit your output. If you want to see real output frequency of your storm topology, check for topology latency for your running storm topology in Storm UI.

Upvotes: 1

Related Questions