CODEWITHSUNDEEP
azureaggregateazure-data-factoryincremental-load
grimleyed
grimleyed

Reputation: 1

Aggregation in Azure Data Flow is Returning Invalid Value

I have created a data flow in Data Factory. Step 1. Read the parquet file. Step 2. Aggregate the file to get the Max(DateField) Step 3. Use a derived column to write in a Value. Step 4. Alter row task with Value and the DateField. Step 5. Sink select the Watermark table to update.

The flow updates the value, but it isn't putting in the max value. The date value is incorrect. Any ideas? Flow_image

Upvotes: 0

Views: 491

Answers (1)

Utkarsh Pal
Utkarsh Pal

Reputation: 4554

max() aggregate function doesn't work on date/string format type. You must pass any column which contains numerical values. Date is not a valid input on which you can apply max function. There is no maximum date term.

Instead you can filter the timestamp and get the latest or oldest date using ADF.

Refer this answer by @Leon to know how to implement the same.

Upvotes: 0

Related Questions