CODEWITHSUNDEEP
python-3.xapache-flinkapache-beam
Metehan Yıldırım
Metehan Yıldırım

Reputation: 401

Apache Beam Python SDK for Windowing with SQL

The problem is, I want to make a windowing inside SqlTransform as

SELECT f_timestamp, line, COUNT(*)
FROM PCOLLECTION
GROUP BY
   line,
   HOP(f_timestamp, INTERVAL '30' MINUTE, INTERVAL '1' HOUR)

My Row transformation mapping is

| "Create beam Row" >> beam.Map(lambda x: beam.Row(f_timestamp= float(x["timestamp_date"]), line = unicode(x["line"])))

And I have an error on the Java side as

Caused by: org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlValidatorException: 
Cannot apply 'HOP' to arguments of type 'HOP(<DOUBLE>, <INTERVAL MINUTE>, <INTERVAL HOUR>)'. 
Supported form(s): 'HOP(<DATETIME>, <DATETIME_INTERVAL>, <DATETIME_INTERVAL>)'

The things I tried:

As I read, Java side uses java.util.Date on the timestamp, how can I work around this issue?

Upvotes: 1

Views: 100

Answers (1)

robertwb
robertwb

Reputation: 5104

You should be able to use apache_beam.utils.timestamp.Timestamp for this.

Upvotes: 1

Related Questions