CODEWITHSUNDEEP
azuresftpazure-data-factory
Aninda Basu
Aninda Basu

Reputation: 183

Azure Data Factory Wildcard Characters

I got a sftp location where generally .csv files are put, and we process or pull the file present there using ADF Copy activity. When no file is present and we give the exact filename and run the pipeline, it fails which is as expected. But when we give a wildcard character such as abc*.csv and run the pipeline with no file present in the sftp location, the copy activity passes though rows written is 0. Can anyone tell me why this happens. We are using Adf v2.

Upvotes: 2

Views: 4390

Answers (1)

MartinJaffer-MSFT
MartinJaffer-MSFT

Reputation: 713

The answer to this is nuanced. Here is the difference:

When you give an exact filename, but the file doesn't exist, Data Factory tries to get it, but the request is returned a 'file not found' error. This is passed up to the activity, and is recognized as a failure.

When you give a wildcard, this is really asking "Get me a list of files that fit this pattern, and then Copy each of them". When there are no files matching the pattern, the result is an empty list. Since the list length is 0, no requests to fetch any file is made, meaning there is no opportunity to be served a 'file not found' error.

This is my reasoning from my experience with Data Factory. I am not a member of the development team.

Upvotes: 2

Related Questions