CODEWITHSUNDEEP
azureazure-pipelinesazure-data-factory
CR7
CR7

Reputation: 99

Azure Data Factory Copy Data From Blob Storage Only New Added file(s)

I would like to create a pipeline with Copy Data (mapping JSON source to MySQL columns). At this moment it is working, but the tool is copying all the files from my source container after the 'created' trigger. Is it possible to retrieve only 'new added' files in my container, so I do not have double data in MySQL?

Example current situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 1 + file 2

Example expected situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 2

Upvotes: 1

Views: 935

Answers (1)

Sally Dabbah
Sally Dabbah

Reputation: 479

This is actually a very used scenario , you should use LastModifiedDate for copying most recently added files and to prevent duplication.

Please check this link : https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

Upvotes: 2

Related Questions