CODEWITHSUNDEEP
azureazure-data-lakeazure-data-factory
Mikel Laburu
Mikel Laburu

Reputation: 157

Azure Data Factory copy between ADLs with a dynamic path

I am trying to create a copy activity between two Azure Data Lakes GEN1. I have to make the copy over a path where one of the subfolders is varible, for example: 

rootFolder/subFolder1/*/subFolder3

where * can take different values, and the copy has to be made automatically for all these possible values, so it was not worth setting that subfolder as a parameter and running the pipeline as many times as the number of possible values.

I would like to know if there is a way to implement this copy activity automatically, I am new to Azure and ADF.

Upvotes: 0

Views: 1304

Answers (1)

Kranthi Pakala
Kranthi Pakala

Reputation: 1422

This can be achieved by using Wildcard filtering in the source settings of your Copy Activity.

Set Wildcard Folder path = rootFolder/subFolder1/*/subFolder3 Wildcard File name = * or *.json or *.txt or *.csv.., etc based on your requirement.

For Example:

enter image description here

To know more about resulting behavior of the folder path and file name with wildcard filters, please refer to this MS Doc: https://learn.microsoft.com/azure/data-factory/connector-azure-data-lake-store#folder-and-file-filter-examples

Here are few threads related to similar requirement which might be helpful:

https://social.msdn.microsoft.com/Forums/en-US/d2b6c77b-c6de-4369-94ba-5a0c7e31a510/iterate-to-every-single-folder-and-each-files-into-a-single-csv?forum=AzureDataFactory#c9696acd-39d0-491f-8ced-7bf135b82c10

https://social.msdn.microsoft.com/Forums/en-US/00c93358-0291-44d9-96d8-266fadc7fd47/copy-data-to-separate-directory-based-on-folder-name?forum=AzureDataFactory

Upvotes: 1

Related Questions