Reputation: 71
I'd like to set up an Azure Data Factory pipeline which performs a move (i.e. copy, verify, delete) operation rather than just a copy operation between Blob Storage and a Data Lake Store. I cannot seem to find any detail on how to do this.
Upvotes: 7
Views: 11366
Reputation: 17111
Just to add a contemporary update for anyone coming across this.
Data Factory V2 has relatively released a dedicated Delete Activity
At the time of writing this supports:
{
"name": "DeleteActivity",
"type": "Delete",
"typeProperties": {
"dataset": {
"referenceName": "<dataset name>",
"type": "DatasetReference"
},
"recursive": true/false,
"maxConcurrentConnections": <number>,
"enableLogging": true/false,
"logStorageSettings": {
"linkedServiceName": {
"referenceName": "<name of linked service>",
"type": "LinkedServiceReference"
},
"path": "<path to save log file>"
}
}
}
Taken from: https://learn.microsoft.com/en-gb/azure/data-factory/delete-activity
Upvotes: 2
Reputation: 14399
Azure Data Factory does not have a built-in activity or option to Move files as opposed to Copy them. You can however do this with a Custom Activity.
This example on github shows how to do this with Azure Blob:
...
blob.DeleteIfExists();
...
https://github.com/Azure/Azure-DataFactory/tree/master/Samples/DeleteBlobFileFolderCustomActivity
If you feel this is an important feature, please add a feedback request:
https://feedback.azure.com/forums/270578-data-factory
A Delete activity has been added recently:
Upvotes: 2
Reputation: 39
From the product team on ADF here. While we're working on "Delete" as a first class activity in ADF, we have published a sample in Github in how users can delete files (in this case, Azure Blob) once they've been copied using ADF copy activity.
https://github.com/Azure/Azure-DataFactory/tree/master/Samples/DeleteBlobFileFolderCustomActivity
This is possible using the ADF custom .Net activity. The sample showcases the following:
Contents of the Github repo:
Upvotes: 0