systemdebt
systemdebt

Reputation: 4951

s3 data lake and Athena - How to conclude the final state

We have devices that can have different sorts of alerts. Alerts can be generated over a period of time so it is time series data.

So let's say:

12:00: Device1 is in the connection alarm stage

12:02: Device alarm acknowledged

12:09: Device alarm off

Now, in reports for devices we will have to say:

when and for how long was device1 in the alarms stage?

-> How should this be stored/queried efficiently from the S3 data lake? I wondering if I should have data partitioned by YYY/MM/DD/ for this. If yes, let's say the user requested the report for 2022-08-01-2022-08-02, and the device alarm was acknowledged on 2022-08-3 and the device alarm had started on 2022-07-31. How would an Athena query find that?

-> If it should not be partitioned by timestamp, I'd still like to know the best way to get information about a device stage for the requested period without missing information for the situations similar to shared above.

Upvotes: 0

Views: 63

Answers (0)

Related Questions