CODEWITHSUNDEEP
azure-data-factory
Luis Jacobo
Luis Jacobo

Reputation: 97

Dynamic filename in Data Factory dataflow source

I’m working with a pipeline that loads table data from onpremise SQL to a datalake csv file dynamically, sinking a .csv file for each table that I already set to load in a versionControl table in a AzureSQL using Foreach.

So, after load the data, i want to update the versionControl table with the lastUpdate date, based on the MAX(lastUpdate) field of each .csv file loaded. To accomplish that, i know that i need to add a dataflow after the copy activity, so i can use the aggregate transformation, but don’t know how to pass the filename to the source of the dataflow dynamically in a parameter.

Thanks!

Upvotes: 1

Views: 3004

Answers (1)

Mark Kromer MSFT
Mark Kromer MSFT

Reputation: 3838

2 options:

  1. Parameterized dataset. Use a source dataset in the dataflow that has a parameter for the file name. You can then pass in that filename as a pipeline parameter.

  2. Parameterized Source wildcard. You can also use a source dataset in the dataflow that points just to a folder in your container. You can then parameterize the wildcard property in the Source and send in the filename there as a pipeline parameter.

Upvotes: 1

Related Questions