CODEWITHSUNDEEP
azure-data-factory
Mancher
Mancher

Reputation: 1

Azure copy data activity

assume that I'm trying to copy 1000 records in a table from a database to an Azure SQL DB/Synapse using ADF Copy activity. if the Copy activity fails after copying 500 records, is it possible to re-run/restart the pipeline such that the Copy activity avoids copying already copied records( 600 records which were copied in the earlier run) and resume copy operation from the remaining 500 records?

Thank you.

n

Upvotes: 0

Views: 187

Answers (1)

Nandan
Nandan

Reputation: 4945

Unfortunately based on my understanding, the copy activity would being from the start and cannot have the scope to copy from where it had failed (in case of bulk copy)

There is a way (though a bad way) wherein you can can use a for each loop with the number of iterations equivalent to the rows and copy each row in an iteration and maintain a watermark feature . In that way in case if copy failed, it would start from the watermark but this would be a bad way and a bad performance as compared to allowing a full rerun

Upvotes: 0

Related Questions