Can we achieve multiple files(placed inside multiple container) copy to different target folder(each for one source container) using Azure datafactory

Question

I am having files in multiple Azure storage containers. And need to copy those files to respective destination folders. Can we achieve this using copy data activity in Azure datafactory ? If yes could you please let me know the steps.

Also needs to implement event based trigger on top it. Can we achieve the whole set up using Azure data factory ?

Answer 1

I don't have any specific copy condition. Simply I need a copy data pipeline. I do have files inside multiple containers which are present inside one storage account in source side. And on destination side I do have a parent folder inside a onprem server and I need to copy all files present inside the storage account to this parent folder with respective hierarchy.

You can use Preserve heirarchy in copy activity sink.

These are my souce containers with some files in a storage account.

Create two binary datasets one for source and another for sink.

Don't give any file path in the source dataset.

In sink dataset, give the file path till your parent folder.

Give the above to copy activity source and sink and in sink, select Preserve heirarchy as copyBehavior.

Containers will be copied with same hierarchy to the target folder like below.

Also needs to implement event based trigger on top it. Can we achieve the whole set up using Azure data factory ?

Yes, this can be done. If you want to copy whole hierarchy every time pipeline triggered, you can add the trigger to the above pipeline.

If you want to copy only the triggered file with same hierarchy in the target you need to use another pipeline for the trigger.

In the storage event trigger, select All containers for the source storage account.

Go through this answer to understand about how to copy the triggered file from source to sink using the dataset parameters to preserve the hierarchy of the file.

But, in this it requires the Blob path ends with. So, in this case you need to create trigger to the same pipeline for every type of file ends with .csv,.json,.. etc to copy all types of files.