CODEWITHSUNDEEP
filepipelineazure-data-factory
Manish Sharma
Manish Sharma

Reputation: 21

ADF-Azure Data factory multiple wild card filtering

I have a condition where i have more than 2 types of files which i have to filter out. I can filter out 1 type using wildcard, something like: *.csv but cant do something like *.xls, *.zip.

I have a pipeline which should convert csv, avro, dat files into .parquet format. But, folder also have .zip, excel, powerpoint files and i want them o be filtered out. Instead of using 3-4 activities i am finding if any way i can use (or) condition to filter out multiple extensions using wildcard option of data factory?

Upvotes: 1

Views: 3045

Answers (1)

Jay Gong
Jay Gong

Reputation: 23792

Dynamic content can't accept multiple wildcards or Regular expression based on my test.

enter image description here

You have to using multiple activities to match the different types of your files.Or you could consider a workaround that using LookUp activity+For-each Activity.

1.LookUp Activity loads all the file names from specific folder.(Child Item)

2.Check the file format in the for-each activity condition.(using endswith built-in feature)

3.If the file format matches the filter condition, then go into the True branch and configure it as dynamic path of dataset in the copy activity.

Upvotes: 1

Related Questions