Azure DataFactory Params - Newbie Question

Question

I'm working with ADF and trying to leverage parameters to make life easier and reduce the number of objects being created in the ADF itself. What I am trying to do, would appear on the surface to be extremely simple, bu in reality its driving me slowly crazy. Would greatly appreciate any assistance!

I am trying to set up a parameterised dataset to be used as a sink target. Inside that dataset I have added a param named "filenames" of type string. In the connection tab I have added that param to the file part of the path. The folder part point to my Azure Data Lake folder and the file part is set to: @dataset().filename which is the result of choosing 'dynamic content' then selecting the param.

So far so good.. my sink target is, as far as I am aware, ready to receive "filenames" to write out to.

This is where it all goes wrong.

I now create a new pipeline. I want to use a list or array of values inside that pipeline which represent the names of the files I want to process. I have been told that I'll need a Foreach to send each of the values one at a time to the COPY DATA task behind the Foreach. I am no stranger to Foreach type loops and behaviors.. but for the life of me I CANNOT see where to set up the list of filenames. I can create a param as a type "array" but how the heck do you populate it?

I have another use case which this problem is preventing me from completing. This use case is, I think, the same problem but perhaps serves to explain the situation more clearly. It goes like this:

I have a linked service to a remote database. I need to copy data from that database (around 12 tables) into the data lake. At the moment I have about 12 "COPY DATA" actions linked together - which is ridiculous. I want to use a Foreach loop to copy the data from source to data lake one after the other. Again, I can set up the sink dataset to be parameterised, just fine... but how the heck do I create the array/list of table names in the pipeline to pass to the sink dataset?

I add the Foreach and inside the foreach a "COPY DATA" but where do I add all the table names?

Would be very grateful for any assistance. THANK YOU.

Answer 1

If you want to manually populate values of an array as a pipeline parameter, you create the parameter with Array type and set the value with syntax like: ["File1","File2","File3"]

You then iterate that array using a ForEach activity.

Inside the ForEach, you reference @item() to get the current file name value the loop is on.

You can also use a Lookup activity to get data from elsewhere and iterate over that using the ForEach.