CODEWITHSUNDEEP
azure-data-factory
Bhaskar
Bhaskar

Reputation: 35

How to Merge files using For each activity in Azure Data Factory

I am using ADF to copy the files from a file server to Azure Blob storage. The files in the directory have the same structure without headers and I need to merge them to a single file in the Blob storage.

enter image description here

I created a ADF pipeline which uses get metadata to fetch the childItems and For each activity to loop through the files one by one

enter image description here

Inside the For each activity there is a copy data activity where I use the file name from the get metadata activity

enter image description here

In the sink setting, I use Merge files as the copy behaviour

enter image description here

When I execute the pipeline, the copy activity gets executed for 3 times and the file in the blob storage gets overwritten with the last file. How do I merge all the 3 files ?

I know we can use a wildcard pattern to select files. Suppose I have 3 files to begin with, when I run the get metadata activity and by the time the control comes to copy job activity and by this time if there is an addition of 4th file in the folder then with the wildcard pattern I will process all the 4 files and get metadata activity gives me the file names of the 3 files which I will use for archiving which is not right

Any help is appreciated

Upvotes: 1

Views: 3154

Answers (1)

Mutaz-MSFT
Mutaz-MSFT

Reputation: 806

You don't need a for each for this. Just one copy activity that Marges all three files.

The trick would be to identify the source files using file path wildcards. if the requirement is to merge all file from source dataset, then merge behaviour in copy activity should be sufficient.

Upvotes: 0

Related Questions