CODEWITHSUNDEEP
timestreamapache-flinkdata-processing
madhairsilence
madhairsilence

Reputation: 3870

Datastream - TimeWindow for past date

I have dataset with past Event Time

01-12-2015 01:10:10
01-12-2015 01:10:20
01-12-2015 01:10:30
01-12-2015 01:10:40
.... (millions of records)

I want to apply timeWindow for this timeWindow(Time.seconds(30))

I can have a TimeExtractor class to get the EventTime in the data. But how do I implement getCurrentWatermark method. It should get the past date and time

Upvotes: 0

Views: 40

Answers (1)

Dawid Wysakowicz
Dawid Wysakowicz

Reputation: 3422

In your case it would be best to use one of the provided TimeStampAssigners see here.

So what I would recommend is something like this:

DataStream<MyEvent> stream = ...

DataStream<MyEvent> withTimestampsAndWatermarks =
stream.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<MyEvent>() {

    @Override
    public long extractAscendingTimestamp(MyEvent element) {
        return element.getCreationTime();
    }
});

Also remember to set proper TimestampCharacteristic:

env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);

Upvotes: 1

Related Questions