CODEWITHSUNDEEP
azurevalidationazure-data-factory
Caleb
Caleb

Reputation: 3

How to validate incoming files in Azure data factory

is there a way to create a template to validate incoming files including such checks as empty file checks, format, data types, record counts along, and will stop the workflow if any of the checks fail. The solution for this requirement should consider multiple file-formats and reduce the burden on ETL processing and checks to enable scale.

File transfer to occur either by trigger or data currency rule

Upvotes: 0

Views: 3967

Answers (2)

Leon Yue
Leon Yue

Reputation: 16431

Data Factory more focus on data transfer, not the file filter.

We could using the get metadata and if-condition to achieve some of the these feature, such as validate the file format, size, file name. You can use Get Metadata to get the file properties and If-condition can help you filter the file.

But that's too complexed for Data Factory to help you achieve all the features you want.

Update:

For example, we can parameter a file in source, :

Create dataset parameter filename and pipeline parameter name: enter image description here

Using Get metadata to get its properties: Item type, Exists, Size, Item name. enter image description here

Output: enter image description here

For example, We can build expression in if-condition to judge if it's empyt(size=0):

@equals(activity('Get Metadata1').output.size,0)

If Ture means it's empty, False no empty. Then we can build the workflow in True or False active.

Hope this helps.

Upvotes: 1

Mark Kromer MSFT
Mark Kromer MSFT

Reputation: 3838

I demonstrate similar techniques to validate source files and take appropriate downstream actions in your pipeline based on those values in this video.

Upvotes: 0

Related Questions