CODEWITHSUNDEEP
azure-data-factoryazure-table-storage
Ahmed ilyas
Ahmed ilyas

Reputation: 5822

Azure Data factory - ignore duplicates in destination?

I am retrieving data from a Sql database and want to insert into an Azure table storage.

However, I want to make sure I am not duplicate inserting data into the table storage.

How can I achieve this using ADF?

The uniqueness for me will be based on at least 2 columns. Let's say...

CustomerNumber SubscriptionNumber

My query from Sql gets the data I need but want to ensure that dups are not inserted in Table storage.

I appreciate the guidance.

Upvotes: 0

Views: 299

Answers (1)

JayashankarGS
JayashankarGS

Reputation: 8140

You have a Merge option in Copy activity sink settings with Azure Table Storage as the sink. Provide a Row key column that serves as a unique identifier.

Below is the sink settings configuration.

enter image description here

Here, I have given:

Insert type : Merge

Row key column : ID

Below is the data in the CSV file and table storage.

In table storage

enter image description here

In CSV file

enter image description here

Now, I have changed the data with the same ID as shown below.

enter image description here

After running the copy activity, the data in table storage is:

enter image description here

and

In table storage

enter image description here

This way, you can have duplicates in the ID column and update new values to corresponding columns.

Unfortunately, you can specify only one column in the row key column.

Upvotes: 0

Related Questions