CODEWITHSUNDEEP
azureazure-blob-storageazure-powershell
Sue Loh
Sue Loh

Reputation: 93

Using Set-AzStorageBlobContent to upload only new content without prompts

I'm enumerating a local folder and uploading to Azure storage. I want to only upload new content to my Azure storage. If I use Set-AzStorageBlobContent with -Force, it'll overwrite everything. If I use it without -Force, it'll prompt on items that already exist. I can use Get-AzStorageBlob to check if the item already exists, but it prints red errors if the item does not exist. I can't find a combination of these items that gracefully uploads only new content without printing any errors or prompting. Am I using the wrong approach?

FINAL EDIT: adding working solution based on suggestions from Ivan Yang. Now only new files are uploaded, without any error messages. The key was to use -ErrorAction Stop to convert the error message into an exception, and then catch the exception.

# In my code this is part of a Test-Blob function that returns $blobFound
$blobFound = $false
try
{
    $blobInfo = Get-AzStorageBlob `
        -Container $containerName `
        -Context $storageContext `
        -Blob $blobPath `
        -ErrorAction Stop

    $blobFound = ($null -ne $blobInfo)
}
catch [Microsoft.WindowsAzure.Commands.Storage.Common.ResourceNotFoundException]
{
    # Eat the error that'd otherwise be printed
}

# Note in my code this is actually a call to my Test-Blob function
if ($false -eq $blobFound)
{
    Set-AzStorageBlobContent `
        -Container $containerName `
        -Context $storageContext `
        -File $sourcePath `
        -Blob $blobPath `
        -Force  # -Force is unnecessary but just being paranoid to avoid prompts
}

Upvotes: 3

Views: 7965

Answers (4)

Yooakim
Yooakim

Reputation: 1817

Not a PowerShell solution but I would suggest that you take a look at AzCopy. It's like RoboCopy but for Azure storage. A command line tool which allows you to synch, copy, move and more. It's free, works on macOS, Linux and Windows. And also, it is fast!

I use AzCopy from PowerShell scripts and it makes lie a lot easier (I'm managing millions of files and the stability and speed of AzCopy really helps)

Upvotes: 1

Ivan Glasenberg
Ivan Glasenberg

Reputation: 29950

I see you have mentioned trying Get-AzStorageBlob, why not use it continually?

The trick here is that you can use try-catch-finally, which can properly handle the error if the blob does not exist in azure.

The sample code works at my side for uploading a single file, and you can modify it to upload multi-files:

$account_name ="xxx"
$account_key ="xxx"      
$context = New-AzStorageContext -StorageAccountName $account_name -StorageAccountKey $account_key    

#use this flag to determine if a blob exists or not in azure. And assume it exists at first.
$is_exist = $true
try
{
 Get-AzStorageBlob -Container test3 -Blob a.txt -Context $context -ErrorAction Stop
}
catch [Microsoft.WindowsAzure.Commands.Storage.Common.ResourceNotFoundException]
{
 #if the blob does not exist in azure, do the following
 $is_exist = $false
 Write-Output "the blob DOES NOT exists."
}
finally
{
 #only execute the code when the blob does not exist in azure blob storage.
 if(!$is_exist)
 {
 Set-AzStorageBlobContent -Container test3 -File "d:\myfolder\a.txt" -Blob a.txt -Context $context
 Write-Output "uploaded!"
 }

}

Upvotes: 3

Martin Thøgersen
Martin Thøgersen

Reputation: 1639

Simply use Set-AzStorageBlobContent -Force all the time.

The alternative is to check for existing file, download the file content, compare the files, and upload if different. The amount of processing/IO will only increase this way.

Upvotes: -1

Thiago Custodio
Thiago Custodio

Reputation: 18387

This command is not smart enough to detect which files are new. You need to keep in the folder just the files you want to upload.

Upvotes: 0

Related Questions