CODEWITHSUNDEEP
apache-stormsliding-window
spitfjre
spitfjre

Reputation: 11

Storm Sliding Window Approach

Currently i try looking into storm for message processing. I find the sliding window feature interesting and tried getting it to work.

But even if i set the interval to say like 5 seconds, the computation behind the window is done much more frequent. It seems with every new message the execute-method for the tuple window gets executed.

builder.setBolt("messageCountBolt",
                new MessageCountBolt()
                        .withWindow(
                                new BaseWindowedBolt.Duration(20, TimeUnit.SECONDS),
                                new BaseWindowedBolt.Duration(5, TimeUnit.SECONDS))
                        .withMessageIdField("id")
                        .withTimestampField("timeStamp")
                        .withLag(new BaseWindowedBolt.Duration(5, TimeUnit.SECONDS)),
                1).globalGrouping("spout");

Someone has an idea why? I want the computation to wait for all messages in the 5 seconds interval.

Upvotes: 1

Views: 749

Answers (2)

kaulmonish
kaulmonish

Reputation: 408

I think the reason is that you are using SlidingWindow - which generates an output for every entry and exit in that window. If you want only one output at the end of the window then you should ideally be using Batch windows or Tumbling windows. To summarise:

  • Sliding window: Keeps each event for the given time window, produce an output whenever new event has added or removed.
  • Batch window: also called tumbling windows, they only produce output at the end of the time window.

Upvotes: 0

Majid Hajibaba
Majid Hajibaba

Reputation: 3260

You must use withTumblingWindow instead withWindow.

withWindow executes on every input tuple and deliver an input batch that contains last input messages. but withTumblingWindow will aggregate all input messages in a batch and delivers whole-in-one.

Upvotes: 2

Related Questions