CODEWITHSUNDEEP
automationazure-data-factory
GuanSe
GuanSe

Reputation: 39

ADF: Using ForEach and Execute Pipeline with Pipeline Folder

I have a folder of pipelines, and I want to execute the pipelines inside the folder using a single pipeline. There will be times when there will be another pipeline added to the folder, so creating a pipeline filled with Execute Pipelines is not an option (well, it is the current method, but it's not very "automate-y" and adding another Execute Pipeline whenever a new pipeline is added is, as you can imagine, a pain). I thought of the ForEach Activity, but I don't know what the approach is.

ADF Pipeline Folder

Upvotes: 1

Views: 1018

Answers (2)

Himanshu Kumar Sinha
Himanshu Kumar Sinha

Reputation: 1786

I have not tried this approach but I think we can use the

  1. ADF RestAPI to get all the details of the pipelines which needs to be executed. Since the response is in JSON you can write it back to temp blob and add filter and focus on what you need .

https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/list-by-factory?tabs=HTTP

  1. You can use the Create RUN API to trigger the pipeline .

https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/create-run?tabs=HTTP

As Joel called out , if different pipeline has different count of paramter , it will be little messy to maintain .

Upvotes: 1

Joel Cochran
Joel Cochran

Reputation: 7738

Folders are really just organizational structures for the code assets that describe pipelines (same for Datasets and Data Flows), they have no real substance or purpose inside the executing environment. This is why pipeline names have to be globally unique rather than unique to their containing folder.

Another problem you are going to face is that the "Execute Pipeline" activity is not very dynamic. The pipeline name has to be known as design time, and while parameter values are dynamic, the parameter names are not. For these reasons, you can't have a foreach loop that dynamically executes child pipelines.

If I were tackling this problem, it would be through an external pipeline management system that you would have to build yourself. This is not trivial, and in your case would have additional challenges because of the folder level focus.

Upvotes: 1

Related Questions