CODEWITHSUNDEEP
azure-data-factory
Maecky
Maecky

Reputation: 2038

Azure Data Factory with Integration Runtime - Delete (or move) file after copy

I have an on premise server with the Microsoft Integration Runtime installed. In Azure Data Factory V2 I created a pipeline that copies files from the on premise server to a blob storage.

After a successful transfer I need to delete the files on the on premise server. I am not able to find a solution for this in the documentation. How can this be achieved?

Upvotes: 1

Views: 1508

Answers (3)

Nicholas
Nicholas

Reputation: 632

If you are simply moving the file then you can use a Binary dataset in a copy activity. This combination makes a checkbox setting visible that when enabled will automatically delete the file once the copy operation completes. This is a little nicer as you do not need the extra delete activity and the file is "moved" only if the copy operation is a success.

enter image description here

Upvotes: 1

Timo Strotmann
Timo Strotmann

Reputation: 391

Recently Azure Data Factory introduced a Delete Activity to delete files or folders from on-premise storage stores or cloud storage stores.

Upvotes: 3

Martin Esteban Zurita
Martin Esteban Zurita

Reputation: 3209

You have the option to call Azure Automation using webhooks, with the web activity. In Azure Automation you can program a powershell or python script with a Hybrid Runbook Worker to delete the file from the on premise server. You can read more on this here: https://learn.microsoft.com/en-us/azure/automation/automation-hybrid-runbook-worker

Another easier option would be to program a script to be run on the server with the windows task scheduler where you run a script to delete the file. Make sure you program the script to be run after data factory has copied the files to the blob, and that's it!

Hope this helped!

Upvotes: 2

Related Questions