Reputation: 177
Within Mapping data flow (trying to avoid Foreach)
Is this possible with a structure like this container/directory/File
Land/’Folder1/20210815’/a.csv
Land/’Folder1/20210816’/b.csv
Land/’Folder2/20210815’/c.csv
I want to load Folder1 and subfiles and sink them to Base container in the data lake
Base/’Folder1/20210815’/a.csv
Base/’Folder1/20210816’/b.csv
In dataflow, I will add one derived column which is loaddate (or anything I want in dataflow).
All files contain the same csv columns structure column1,column2,column3,loaddate
I do not want to specify a date basically bulk load any csv under a subfolder
Thanks
Upvotes: 0
Views: 760
Reputation: 1145
Here is my example using a Copy Activity without using a For Each:
I have 4 files with this structure in Blob storage:
Folder1/20210815/Placeholder.csv
Folder1/20210815/Placeholder5.csv
Folder1/20210816/Placeholder3.csv
Folder2/20210816/Placeholder3.csv
Here is the copy data details:
Here are those files and as you can see the subfolders were copied over. This theoretically should work in Data Lake, but I can't be sure. It definitely works in Azure Blob Storage.
Here are the contents of the 20210815 Subfolder:
Upvotes: 1