Parameterise Sink and Source Option in DataFlow

Question

I am loading the data source raw storage container which is single directory, we have multiple subfolders, each folder like one tables. so that we are loading the data and put back to another stage container. we have to make the dynamic source option path and sink dataset container path and subfolders. how do we handle this usecase in adf. please suggest if you have any forum links, please share to us.

source options as below like,

container/test1subfolder/.csv files

container/test2subfolder/.csv files

sink datasets with processed files like as below

stagecontainer/test1subfolder/.csv files

stagecontainer/test2subfolder/.csv files

we want to make this dynamic read subfolder and write dynamic subfolder. each subfolder under we have to maintain the .csv files

please suggest.

Answer 1

If you are using Dataflow it is a complicated process.

Instead of Dataflow you can use Copy activity and then make it a dynamically read subfolder with Wild card as shown in the below image.

Output: