Filter out file using wildcard path azure data factory

Question

I am working on a pipeline and while using the copy activity, in the file wildcard path I would like to skip a certain file and only copy the rest. Is there an expression for that ? I know that a * is used to match zero or more characters but in this case, I would like an expression to skip a certain file.

I am still very new to adf

Thank you.

Answer 1

In Get Metadata activity, we can add an expression to get files of a specific pattern. I tried to write an expression to exclude files but was not successful.

Below is what I have tried to exclude/skip a file from the list of files to process.

Files in the Input folder:

Azure Data Factory Pipeline:

1. In my Input folder, I have 2 types of files .csv and .txt. You can add expression in the filename to get the only “.csv” files using Get Metadata activity

Filename Expression: *.csv

Get Metadata activity Output:

2. Connect filter activity to Get Metadata activity output to exclude a file (or files of specific pattern) from the list of files from child items.

Filter activity Settings:

Items: @activity('Get Metadata1').output.childitems

Condition: @not(contains(item().name,'1c56d6s4s33s4_Sales_09112021.csv'))

Output of Filter activity:

3. Process each value of filter activity using foreach activity.

Items: @activity('Filter1').output.Value

This loop runs 2 times as there are only 2 files that returned from filter activity output after excluding a file.

Answer 2

I'm not sure you can use the wildcard feature to skip a specific file, unless all the other files follow a pattern the exception does not follow.

One approach would be to use GetMetadata to list the files:

Note the inclusion of the "ChildItems" field, this will list all the items (Folders and Files) in the directory.

Next, use a Filter activity to reference only the files:

Items code:

@activity('Get Child Items').output.childItems

Filter code:

@and(equals(item().type,'File'),endswith(item().name,'.txt'))

NOTE: This example filters to Files with a .txt extension. You would change this code to meet your criteria.

Finally, use a ForEach to loop over the now filtered items. The ForEach would contain our COPY activity for each individual item: