CODEWITHSUNDEEP
powershellazure-blob-storage
Eggman
Eggman

Reputation: 23

How to go through Azure Blobs without memory problems

I am trying to convert the access tier of our blobs in Azure storage from Cool to Archive. I am using a Powershell script for this, but once i get to some of our containers with =200k files, it sucks out all of my RAM.

I had hoped that making a Foreach-obejct{} after the Get-AzStorageBlob would make it run through the object right after so that it wouldnt store it.

#Account information
$StorageAcc = "StorageAccount-info"
$StorageAccKey = "StorageAccKey-info"
$containername = "container1"

#Create a storage context
$context = New-AzStorageContext -StorageAccountName $StorageAcc -StorageAccountKey $StorageAccKey 

Get-AzStorageBlob -Container $containerName -Context $context | ForEach-Object {$_.ICloudBlob.SetStandardBlobTier("Archive")}

I expect this to run through a container with x amount of files without the pc crashing because of memory issues, time is not an issues.

Right now i can run thorugh 132k in 90 mins, that takes about 400Mb of my memory/RAM. Thank you!

EDIT: For anybody how could need something like this. It will go through all the blobs and check if its has already been archived, this saves alot of time. It will also save the ContinuationToken in a text file,so that if something happens you dont have to start over. Remember to edit the path for the file

$StorageAcc = "StorageAccount-info"
$StorageAccKey = "StorageAccKey-info"
$containername = "container1"
$token = $null
$total = 0
$start = 0

#Create a storage context
$context = New-AzStorageContext -StorageAccountName $StorageAcc -StorageAccountKey $StorageAccKey 
do{

$blobs = Get-AzStorageBlob -Container $containerName -Context $context -MaxCount 10000 -ContinuationToken $token
$Total += $Blobs.Count
    foreach ($blob in $blobs)
    {
        If($blob.ICloudBlob.Properties.StandardBlobTier -eq "cool"){

        $blob.ICloudBlob.SetStandardBlobTier("Archive")
        }

    }

if($Blobs.Length -le 0) { Break;}
    $Token = $Blobs[$blobs.Count -1].ContinuationToken;
    Echo "Total $Total blobs in container $ContainerName"
    add-content C:\Users\Admin\Desktop\Token.txt $Token.NextMarker
        if($start -eq 0)
        {
        $start += 1
        $token.NextMarker = "Token"
        }
}
 While ($Token -ne $Null)
 Echo "Total $Total blobs in container $ContainerName the end"

Upvotes: 2

Views: 829

Answers (1)

lloyd
lloyd

Reputation: 1811

It's hinted at in the documentation. You process them in batches. With some minor modification to their code this should work. Set the max return to what you need it to be.

$MaxReturn = 10000
$ContainerName = "abc"
$Total = 0
$Token = $Null
do
{
    $Blobs = Get-AzStorageBlob -Container $ContainerName -MaxCount $MaxReturn  -ContinuationToken $Token
    $Total += $Blobs.Count
    if($Blobs.Length -le 0) { Break;}

    foreach($blob in $blobs) {
        $blob.ICloudBlob.SetStandardBlobTier("Archive")
    }


    $Token = $Blobs[$blobs.Count -1].ContinuationToken;
}
While ($Token -ne $Null)
Echo "Total $Total blobs in container $ContainerName archived"

Upvotes: 2

Related Questions