CODEWITHSUNDEEP
azureblobazure-blob-storage
Jerry
Jerry

Reputation: 23

What is the optimal way of setting Blob index tag in Azure Blob storage for large amount of data

I need to add blob index tags to a large amount of data existing in azure blob storage (nearly 40TB). The same should be done to all blobs that will be uploaded in future. For the latter ,I need to create a periodically executing powershell script which should only get the blobs with last modified date greater than a given date time and add a index tag with value equal to last modified date . Is this possible ? If yes , How ? . Also what is the ideal way to add index to large amount of data with least performance impact?
NB : I'm new to azure . Better ideas are appreciated.

My objective is to do a backup of the blobs periodically, say weekly. Right now, due to the large blob size, it is taking more than 1 day for iterating through all of them. I'm hoping to reduce this time leveraging index tags and for achieving this I do not want to set the index tag during blob upload.(It's a sort of 'don't touch' legacy code). To summarize ,I need a mechanism to quickly find the blobs added within last week (without iterating through all blobs), set index tags with the blob's last modified date and some time in future ,I will do back up by filtering on these index tags.

Upvotes: 2

Views: 2499

Answers (1)

SumanthMarigowda-MSFT
SumanthMarigowda-MSFT

Reputation: 2354

Could you please clarify what are you trying to accomplish by adding the Last-Modified time as an indexed tag? How are you planning to use this tag?

How many blobs do you have in this storage account?

Blob index tags can be set using the Set Blob Tags API, or equivalent PS/CLI such as Set-AzStorageBlobTag.

For existing blobs, you’ll need to call Get Blob Properties to extract the Last-Modified time and then set that as a tag.

For new blobs uploaded to the storage account, do you have control over the clients that are uploading the blobs? If so, it would be more efficient if the client set the current time as a tag when uploading the blob. You can force clients to include tags with a particular key during upload using ABAC – see Example Azure role assignment conditions (preview) - Azure RBAC - Azure Storage | Microsoft Docs.

Upvotes: 2

Related Questions