CODEWITHSUNDEEP
azure-data-factoryazure-data-lake
Peter Retep
Peter Retep

Reputation: 45

Its possible to remove files from data lake using data factory?

Its possible remove files from datalake using data factory? Its very dificult to do that, i m stuck at it.

Upvotes: 1

Views: 226

Answers (2)

Jorge Ribeiro
Jorge Ribeiro

Reputation: 1138

The best option you have is create an Azure Batch to do that.

https://learn.microsoft.com/en-us/azure/batch/

You can create an application that remove the files and call it from data factory. You can pass the files as parameteres too.

Upvotes: 0

DraganB
DraganB

Reputation: 1138

It's not possible to delete a file using Data Factory. I found that the main purpose of Data Factory is ETL orchestration.

If you are familiar with .NET/Java coding, I would suggest you to create an Azure Function that will be called from ADF and it will delete a file. Azure Functions are now supported in ADF (don't need to use HTTP trigger/web component)

There is an SDK for File operation on Data Lake, and it contains method Delete, that deletes a file from Data Lake Store. Take a look on this docs: https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-data-operations-net-sdk#see-also

Delete method: https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.datalake.store.adlsclient.delete?view=azure-dotnet

You can pass path to the file as a request parameter/header whatever you feel comfortable from ADF to Azure Function. Hope this suggestion can work for you.

Upvotes: 1

Related Questions