Nested forEach activity in Azure Data factory V2

Question

I am trying to use two forEach activities to iterate on subfolders of folders with parameters to get metadata of subfolders. I have forEach1 and forEach2 with their own items array. Within the second for loop I need to combine both for loops' item() in a Metada activity to access my dataset like @item1()@item2(). Is this possible?

Answer 1

I reproduced this issue. Please find below the detailed explanation.

Scenario: In the RAW container, within the main folder 'manager,' there are some '.txt' files and several subfolders such as 'employee1,' 'employee2,' 'employee3,' etc. Each of these subfolders also contains several '.txt' files.

Scope: Move the files from the main folder and the sub folders to a different container 'Curated' using ADF pipeline.

Issue: To read the files in the main folder, we use a 'For Each' activity. However, to read the files inside the subfolder, we need another 'For Each' activity. Unfortunately, nested 'For Each' activities are not allowed in ADF.

Workaround: Use an 'Execute pipeline' activity inside the foreach activity to iterate the subfolder and to read the Sub Folder files.

Step 1: Since nested For Each activity is not supported in ADF, we need to have 2 pipelines (master and child). In the master pipeline add a 'Get Metadata' activity read the files and folder present in the main folder. Add the Argument as 'Child Items' in order to read all the items.

Step 2: Add 2 filter activities. First filter activity is to filter the files and the second filter activity is to filter the folders.

Items: @activity('Get Metadata1').output.childItems

Condition to read txt files: @and(equals(item().type,'File'),endswith(item().name,'.txt'))

Condition to read folders: @equals(item().type,'Folder')

Step 3: Add a 'For each' activity to the output of the Filter Activity to iterate over the txt files and inside the 'For Each' activity add a 'Copy' activity copy the files from Raw container to the Curated container.

'For each' activity Settings: @activity('Filter1').output.value

Add the source and sink settings. Copy activity source setting:

Step 4: Create a child pipeline by adding 'Get Metadata' activity to read the sub Folder and connect the output of the 'Get Metadata' activity to 'For each' to iterate over the files present in the Sub Folder.

Overall Child pipeline design:

'Get Metadata' activity settings:

Child pipeline Copy activity:

Step 5: Add a 'For each' activity to the output of the second Filter Activity to iterate over the Sub Folders. Inside the 'For Each' activity add an 'Execute Pipeline' activity to iterate over the subfolders.

Overall master pipeline design:

Hope the below steps are helpful on how to add nested 'For Each' in ADF.

Answer 2

Nested foreach activity is not allowed. But you could use an execute pipeline activity inside the foreach activity. And in the nested pipeline, you could have another foreach.

Answer 3

It is possible but the second ForEach activity needs to be inside the first one, not another activity in the pipeline.

As you have it now, the first ForEach will run until completion, then the second one will start and you cannot access the items in the first one.

Hope this helped!