CODEWITHSUNDEEP
azureetlazure-data-factory
Luiz Fernandes
Luiz Fernandes

Reputation: 141

Data Factory - Use wildcard to determine filename

I'm using Data Factory in a way to move a file from my file system to a Blob Storage.

The problem is: the file is named using YYYYMMDD identifiers, [which I could resolve by partitioning ({year}{month}{day})] AND TIME (that is the file creation time).

How do I get the filename having the dynamic time in it? I already tried using *, _ and other usual wildcards.

I will really appreciate your help!

Upvotes: 1

Views: 3013

Answers (1)

Paul Andrew
Paul Andrew

Reputation: 3253

In ADFv1 the combinations of file filter, file name and file path are really limited and the wild cards don't work together across the different attributes.

Given your situation I think there isn't currently anything you'll be able to do about this.

Best options I can suggest:

  1. Get the year month and day created as parent folders in the location where the source data comes from.
  2. Ignore the file name totally. Then use a custom activity shred the file names with some C# in a downstream activity.

I done option 2 in the past as changing source systems isn't always an option.

I appreciate neither of these options are ideal. Option 3 would be to do something more dynamically within an ADFv2 activity using expressions for the file path.

Hope this helps.

Upvotes: 1

Related Questions