Bilal Shafqat
Bilal Shafqat

Reputation: 821

How to use wildcards in filename in AzureDataFactoryV2 to copy only specific files from a container?

So I have a pipeline in AzureDataFactoryV2, in that pipeline I have defined a copyActivity to copy files from blob storage to Azure DataLake Store. But I want to copy all the files except the files that have "-2192-" string in them.

So If I have these files:

213-1000-aasd.csv
343-2000-aasd.csv
343-3000-aasd.csv
213-2192-aasd.csv

I want to copy all using copyactivity but not 213-2192-aasd.csv. I have tried using different regular expression in wildcard option but no success.

According to my knowledge regular expression should be: [^-2192-].csv

But it gives errors on this. Thanks.

Upvotes: 0

Views: 525

Answers (1)

Joel Cochran
Joel Cochran

Reputation: 7728

I don't know whether the data factory expression language supports Regex. Assuming it does not, the Wildcard is probably positive matching only, so Wildcard to exclude specific patterns seems unlikely.

What you could do is use 1) Get Metadata to get the list of objects in the blob folder, then 2) a Filter where the item().type is 'File' and index of '-2192-' in the file name is < 0 [the indexes are 0-based], and finally 3) a ForEach over the Filter that contains the Copy activity.

Upvotes: 1

Related Questions