CODEWITHSUNDEEP
azureazure-eventhubazure-stream-analyticsazure-blob-storage
Andrea Cattaneo
Andrea Cattaneo

Reputation: 588

How does time scan work in Azure Stream Analytics tumbling window and hopping window?

I've an Event Hub in Azure Cloud that takes messages where I have a timestamp value and other parameters. The timestamp is aligned to stream analytics using the command 

TIMESTAMP AS [TimeStamp]

This is the Stream Analytics query (the input is the Event Hub, the output in this case is a blob)

SELECT
    DateAdd(minute, -1, System.Timestamp) as FromTimestamp, System.Timestamp as ToTimestamp, 
    [MachineType], [MachineNumber], [Part], [PartNumber], [ValueKind], AVG(Value) AS AverageValue
INTO
    [blob-avg]
FROM
    [input]
TIMESTAMP BY [TimeStamp]
WHERE [ValueKind]='RPM' OR [ValueKind]='CUR' OR [ValueKind]='POW'
GROUP BY [MachineType], [MachineNumber], [Part], [PartNumber], [ValueKind], SlidingWindow(minute, 1)

I think that the timestamp of the message will be considered as the timestamp to compare but, how it is scanned? At the Utc time? Let me say, in the message I've a timestamp 12:00 (GMT+2), and the UTC now is 10.00 Does the tumbling consider the data arrived 2 hours ago instead of the actual? (at timestamp 10:00 (GMT+2)) (actually it seems to me something like that).

And what append if a message arrives with a delay greater than the 2 hours? let me say that a message arrives with one day of delay, will the tumbling be recalcoulated?

Upvotes: 1

Views: 899

Answers (1)

Vignesh Chandramohan
Vignesh Chandramohan

Reputation: 1306

[Timestamp] column will be converted to datetime, if the format was GMT, it will be taken into account and it is safe to assume that everything will be converted to UTC when time related calculations are made.

Azure Stream Analytics is continuously reading data from the source. And late arrival policy + window decides how to handle late events.

Please have a look at https://msdn.microsoft.com/en-us/library/azure/mt674682.aspx

and

https://blogs.msdn.microsoft.com/streamanalytics/2015/05/17/out-of-order-events/

for more details about out of order policies.

For your specific example, if the message arrives with a delay of greater than 2 hours and your late arrival policy is to drop events, the events will be dropped. If it is adjust, the timestamp will be adjusted to current processing time.

Upvotes: 1

Related Questions