CODEWITHSUNDEEP
pythongoogle-cloud-dataflowapache-beamapache-beam-io
zzob
zzob

Reputation: 1134

Apache Beam windowing for day

I would like to extract data using windows function on apache beam, by day timeframe. Which I worked on python and used FixedWindow for capture the data.

And I had problem about consistency of data cause this code is working by count duration timestamp,

beam.WindowInto(window.FixedWindows(1440*60)) # minute of whole day * second

So that mean that if I start beam pipeline at June-3 3:00PM, It's will be end at June-4 3:00PM.

I want something like, If I have to start the pipeline at June-3 3:00PM, When the time has arrivel to June-4 0:00AM,
The windows function should start new capture, After June-3 11:59:59 PM

so anyone have idea? or the windows function didn't has supported kind of this work.

Upvotes: 0

Views: 694

Answers (1)

danielm
danielm

Reputation: 3010

The windows are not based on the start time of the pipeline, they are based off of the Unix epoch.

In your case, if you want the windows to be aligned days, you can use CalendarWindows. You'll just need to specify the time zone in which the days should be measured.

Upvotes: 1

Related Questions