CODEWITHSUNDEEP
data-cleaningazure-data-factoryazure-blob-storage
Ehsan Mirsaeedi
Ehsan Mirsaeedi

Reputation: 7592

Azure Data Factory: Data Lifecycle Management and cleaning up stale data

I'm working on a requirement to reduce the cost of data storage. It includes the following tasks:

  1. Being able to remove files from File Share and blobs from Blob Storage, based on their last modified date.
  2. Being able to change the tier of individual blobs, based on their last modified date.

Does Azure Data Factory has built-in activities to take care of these tasks? What's the best approach for automating the clean-up process?

Upvotes: 0

Views: 476

Answers (1)

Jay Gong
Jay Gong

Reputation: 23792

1.Being able to remove files from File Share and blobs from Blob Storage, based on their last modified date.

This requirement could be implemented by ADF built-in method: Delete Activity.

Please create a blob storage dataset and just refer to this example and configure the range of last modify date :https://learn.microsoft.com/en-us/azure/data-factory/delete-activity#clean-up-the-expired-files-that-were-last-modified-before-201811

Please consider some back up strategy for some accidents because: enter image description here

2.Being able to change the tier of individual blobs, based on their last modified date.

No built-in feature to complete this in ADF. However,while i notice that your profile shows you are .net maker, so follow this case:Azure Java SDK - set block blob to cool storage tier on upload so that you could know the Tier could be changed in sdk code. That's easy to create an Azure Function to do such simple task. Moreover,ADF supports Azure Function Activity.

Upvotes: 1

Related Questions