CODEWITHSUNDEEP
jsonazureloopsforeachazure-data-factory
Regazzi
Regazzi

Reputation: 115

Azure Data Factory For Each Avoid Array

Im quit new in ADF so here's the challenge from me.

I have a Pipeline that consist a LookUp activity and ForEach and inside this a Copy Activity

enter image description here

enter image description here

enter image description here

enter image description here

When i run this pipeline the first output of the Lookup activity looks like this

enter image description here

The output contains 11 different values. From my perspective i only see 11 records that will need to be copied to my Sink which is Azure SQL DB.

The input of the ForEach activity looks like this

enter image description here

During the running the Pipeline copy 11 times and in my sql database it has now 121 records. This amount is based on 11 rows multiple 11 iteration. This is not the output which i expected. I only expect 11 rows in my sink table. How can i change this pipeline in order to achieve the expected outcome of only 11 rows?

Many thanks!

Upvotes: 0

Views: 436

Answers (1)

Aswin
Aswin

Reputation: 7116

In order to copy data, Lookup activity and copy data source activity should not be given same configuration. If given so, duplicate rows will be copied. I tried to repro the same in my environment.

enter image description here

If 3 records are there in source data, 3 times 3 records will be copied.

enter image description here

enter image description here

  • In order to avoid duplicates, we can use only copy activity to copy data from Source to sink.

enter image description here

  • Only 3 records are in target table.

enter image description here

Upvotes: 1

Related Questions