gabi
gabi

Reputation: 1376

Nested forEach activity in Azure Data factory V2

I am trying to use two forEach activities to iterate on subfolders of folders with parameters to get metadata of subfolders. I have forEach1 and forEach2 with their own items array. Within the second for loop I need to combine both for loops' item() in a Metada activity to access my dataset like @item1()@item2(). Is this possible?

enter image description here

Upvotes: 6

Views: 14163

Answers (3)

Faitus Joseph
Faitus Joseph

Reputation: 69

I reproduced this issue. Please find below the detailed explanation.

Scenario: In the RAW container, within the main folder 'manager,' there are some '.txt' files and several subfolders such as 'employee1,' 'employee2,' 'employee3,' etc. Each of these subfolders also contains several '.txt' files.

Scope: Move the files from the main folder and the sub folders to a different container 'Curated' using ADF pipeline.

Issue: To read the files in the main folder, we use a 'For Each' activity. However, to read the files inside the subfolder, we need another 'For Each' activity. Unfortunately, nested 'For Each' activities are not allowed in ADF.

Workaround: Use an 'Execute pipeline' activity inside the foreach activity to iterate the subfolder and to read the Sub Folder files.

enter image description here

Step 1: Since nested For Each activity is not supported in ADF, we need to have 2 pipelines (master and child). In the master pipeline add a 'Get Metadata' activity read the files and folder present in the main folder. Add the Argument as 'Child Items' in order to read all the items.

enter image description here

Step 2: Add 2 filter activities. First filter activity is to filter the files and the second filter activity is to filter the folders.

Items: @activity('Get Metadata1').output.childItems

Condition to read txt files: @and(equals(item().type,'File'),endswith(item().name,'.txt'))

Condition to read folders: @equals(item().type,'Folder')

enter image description here

Step 3: Add a 'For each' activity to the output of the Filter Activity to iterate over the txt files and inside the 'For Each' activity add a 'Copy' activity copy the files from Raw container to the Curated container.

enter image description here

'For each' activity Settings: @activity('Filter1').output.value

Add the source and sink settings. Copy activity source setting:

enter image description here

Step 4: Create a child pipeline by adding 'Get Metadata' activity to read the sub Folder and connect the output of the 'Get Metadata' activity to 'For each' to iterate over the files present in the Sub Folder.

Overall Child pipeline design:

enter image description here

'Get Metadata' activity settings:

enter image description here

Child pipeline Copy activity: enter image description here

Step 5: Add a 'For each' activity to the output of the second Filter Activity to iterate over the Sub Folders. Inside the 'For Each' activity add an 'Execute Pipeline' activity to iterate over the subfolders.

enter image description here

Overall master pipeline design:

enter image description here

Hope the below steps are helpful on how to add nested 'For Each' in ADF.

Upvotes: 0

Fang Liu
Fang Liu

Reputation: 2363

Nested foreach activity is not allowed. But you could use an execute pipeline activity inside the foreach activity. And in the nested pipeline, you could have another foreach.

Upvotes: 9

Martin Esteban Zurita
Martin Esteban Zurita

Reputation: 3209

It is possible but the second ForEach activity needs to be inside the first one, not another activity in the pipeline.

As you have it now, the first ForEach will run until completion, then the second one will start and you cannot access the items in the first one.

Hope this helped!

Upvotes: 0

Related Questions