CODEWITHSUNDEEP
c#azureazure-blob-storagecache-control
Gavin
Gavin

Reputation: 5871

Add Cache-Control and Expires headers to Azure Storage Blobs

I'm using Azure Storage to serve up static file blobs but I'd like to add a Cache-Control and Expires header to the files/blobs when served up to reduce bandwidth costs.

Application like CloudXplorer and Cerebrata's Cloud Storage Studio give options to set metadata properties on containers and blobs but get upset when trying to add Cache-Control.

Anyone know if it's possible to set these headers for files?

Upvotes: 42

Views: 33450

Answers (9)

JonT
JonT

Reputation: 95

Here is a batch/unix script for everyone that does not sit on a Windows machine with PowerShell. The following script loops through all blobs and sets the Content-Cache property (Cache-Control http header) on the blobs individually.

Unfortunately, there is no good is no way to set properties on several blobs simultaneously, so this is a time consuming task. It usually takes around 1–2 seconds per blob. However, as Jay Borseth points out, the process is significantly accelerated if run it from a server in the same data center as your storage account.

# Update Azure Blob Storage blob's cache-control headers
# /content-cache properties
# 
# Quite slow, since there is no `az storage blob update-batch`
#
# Created by Jon Tingvold, March 2021
#
#
# If you want progress, you need to install pv:
# >>> brew install pv  # Mac
# >>> sudo apt install pv  # Ubuntu
#

set -e  # exit when any command fails

AZURE_BLOB_CONNECTION_STRING='DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=XXXXXXXXXXXX;AccountKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=='
CONTAINER_NAME=main

BLOB_PREFIX='admin/'
CONTENT_CACHE='max-age=3600'
NUM_RESULTS=10000000  # Defaults to 5000

BLOB_NAMES=$(az storage blob list --connection-string $AZURE_BLOB_CONNECTION_STRING --container-name $CONTAINER_NAME --query '[].name' --output tsv --num-results $NUM_RESULTS --prefix $BLOB_PREFIX)
NUMBER_OF_BLOBS=$(echo $BLOB_NAMES | wc -w)

echo "Ask Azure for files in Blob Storage ..."
echo "Set content-cache on $NUMBER_OF_BLOBS blobs ..."

for BLOB_NAME in $BLOB_NAMES
do
  az storage blob update --connection-string $AZURE_BLOB_CONNECTION_STRING --container-name $CONTAINER_NAME --name $BLOB_NAME --content-cache $CONTENT_CACHE > /dev/null;
  echo "$BLOB_NAME"

# If you don't have pv install, uncomment  everything after done
done | cat | pv -pte --line-mode --size $NUMBER_OF_BLOBS > /dev/null

Upvotes: 1

Jay Borseth
Jay Borseth

Reputation: 2003

Here's an updated version of Joel Fillmore's answer using Net 5 and V12 of Azure.Storage.Blobs. (Aside: wouldn't it be nice if default header properties could be set on the parent container?)

Instead of creating a website and using a WorkerRole, Azure has the ability to run "WebJobs". You can run any executable on demand on a website at the same datacenter where your storage account is located to set cache headers or any other header field.

  1. Create a throw-away, temporary website in the same datacenter as your storage account. Don't worry about affinity groups; create an empty ASP.NET site or any other simple site. The content is unimportant. I needed to use at least a B1 service plan, otherwise the WebJob aborted after 5 minutes.
  2. Create a console program using the code below which works with the updated Azure Storage APIs. Compile it for release, and then zip the executable and all required DLLs into a .zip file, or just publish it from VisualStudio and skip #3 below.
  3. Create a WebJob and upload the .zip file from step #2.
  4. Run the WebJob. Everything written to the console is available to view in the log file created and accessible from the WebJob control page. enter image description here
  5. Delete the temporary website, or change it to a Free tier (under "Scale Up").

The code below runs a separate task for each container, and I'm getting up to 100K headers updated per minute (depending on time of day?). No egress charges.

using Azure;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace AzureHeaders
{
    class Program
    {
        private static string connectionString = "DefaultEndpointsProtocol=https;AccountName=REPLACE_WITH_YOUR_CONNECTION_STRING";
        private static string newCacheControl = "public, max-age=7776001"; // 3 months
        private static string[] containersToProcess = { "container1", "container2" };

        static async Task Main(string[] args)
        {
            BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);

            var tasks = new List<Task>();
            foreach (var container in containersToProcess)
            {
                BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(container);
                tasks.Add(Task.Run(() => UpdateHeaders(containerClient, 1000)));  // I have no idea what segmentSize should be!
            }
            Task.WaitAll(tasks.ToArray());
        }

        private static async Task UpdateHeaders(BlobContainerClient blobContainerClient, int? segmentSize)
        {
            int processed = 0;
            int failed = 0;
            try
            {
                // Call the listing operation and return pages of the specified size.
                var resultSegment = blobContainerClient.GetBlobsAsync()
                    .AsPages(default, segmentSize);

                // Enumerate the blobs returned for each page.
                await foreach (Azure.Page<BlobItem> blobPage in resultSegment)
                {
                    var tasks = new List<Task>();

                    foreach (BlobItem blobItem in blobPage.Values)
                    {
                        BlobClient blobClient = blobContainerClient.GetBlobClient(blobItem.Name);
                        tasks.Add(UpdateOneBlob(blobClient));
                        processed++;
                    }
                    Task.WaitAll(tasks.ToArray());
                    Console.WriteLine($"Container {blobContainerClient.Name} processed: {processed}");
                }
            }
            catch (RequestFailedException e)
            {
                Console.WriteLine(e.Message);
                failed++;
            }
            Console.WriteLine($"Container {blobContainerClient.Name} processed: {processed}, failed: {failed}");
        }

        private static async Task UpdateOneBlob(BlobClient blobClient) {
            Response<BlobProperties> propertiesResponse = await blobClient.GetPropertiesAsync();
            BlobHttpHeaders httpHeaders = new BlobHttpHeaders
            {
                // copy any existing headers you wish to preserve
                ContentType = propertiesResponse.Value.ContentType,
                ContentHash = propertiesResponse.Value.ContentHash,
                ContentEncoding = propertiesResponse.Value.ContentEncoding,
                ContentDisposition = propertiesResponse.Value.ContentDisposition,
                // update CacheControl
                CacheControl = newCacheControl  
            };
            await blobClient.SetHttpHeadersAsync(httpHeaders);
        }
    }
}

Upvotes: 13

Stephen McDowell
Stephen McDowell

Reputation: 959

Here's an updated version of Joel Fillmore's answer consuming WindowsAzure.Storage v9.3.3. Note that ListBlobsSegmentedAsync returns a page size of 5,000 which is why the BlobContinuationToken is used.

    public async Task BackfillCacheControlAsync()
    {
        var container = await GetCloudBlobContainerAsync();
        BlobContinuationToken continuationToken = null;

        do
        {
            var blobInfos = await container.ListBlobsSegmentedAsync(string.Empty, true, BlobListingDetails.None, null, continuationToken, null, null);
            continuationToken = blobInfos.ContinuationToken;
            foreach (var blobInfo in blobInfos.Results)
            {
                var blockBlob = (CloudBlockBlob)blobInfo;
                var blob = await container.GetBlobReferenceFromServerAsync(blockBlob.Name);
                if (blob.Properties.CacheControl != "public, max-age=31536000")
                {
                    blob.Properties.CacheControl = "public, max-age=31536000";
                    await blob.SetPropertiesAsync();
                }
            }               
        }
        while (continuationToken != null);
    }

    private async Task<CloudBlobContainer> GetCloudBlobContainerAsync()
    {
        var storageAccount = CloudStorageAccount.Parse(_appSettings.AzureStorageConnectionString);
        var blobClient = storageAccount.CreateCloudBlobClient();
        var container = blobClient.GetContainerReference("uploads");
        return container;
    }

Upvotes: 2

John Washam
John Washam

Reputation: 4103

Sometimes, the simplest answer is the best one. If you just want to manage a small amount of blobs, you can use Azure Management to change the headers/metadata for your blobs.

  1. Click on Storage, then click on the storage account name.
  2. Click the Containers tab, then click on a container.
  3. Click on a blob, then click on Edit at the bottom of the screen.

In that edit window, you can customize the Cache Control, Content Encoding, Content Language, and more.

Note: you cannot currently edit this data from the Azure Portal

Upvotes: 3

frank tan
frank tan

Reputation: 141

Set storage blob cache-control Properties by PowerShell script

https://gallery.technet.microsoft.com/How-to-set-storage-blob-4774aca5

#creat CloudBlobClient 
Add-Type -Path "C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\v2.3\ref\Microsoft.WindowsAzure.StorageClient.dll" 
$storageCredentials = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey -ArgumentList $StorageName,$StorageKey 
$blobClient =   New-Object Microsoft.WindowsAzure.StorageClient.CloudBlobClient($BlobUri,$storageCredentials) 
#set Properties and Metadata 
$cacheControlValue = "public, max-age=60480" 
foreach ($blob in $blobs) 
{ 
  #set Metadata 
  $blobRef = $blobClient.GetBlobReference($blob.Name) 
  $blobRef.Metadata.Add("abcd","abcd") 
  $blobRef.SetMetadata() 

  #set Properties 
  $blobRef.Properties.CacheControl = $cacheControlValue 
  $blobRef.SetProperties() 
}

Upvotes: 0

Tekz
Tekz

Reputation: 1339

This might be too late to answer, but recently I wanted to do the same in different manner, where I have list of images and needed to apply using powershell script (of course with the help of Azure storage assembly) Hope someone will find this useful in future.

Complete explanation given in Set Azure blob cache-control using powershell script

Add-Type -Path "C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\v2.3\ref\Microsoft.WindowsAzure.StorageClient.dll"

$accountName = "[azureaccountname]"
$accountKey = "[azureaccountkey]"
$blobContainerName = "images"

$storageCredentials = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey -ArgumentList $accountName,$accountKey
$storageAccount = New-Object Microsoft.WindowsAzure.CloudStorageAccount -ArgumentList $storageCredentials,$true
#$blobClient = $storageAccount.CreateCloudBlobClient()
$blobClient =  [Microsoft.WindowsAzure.StorageClient.CloudStorageAccountStorageClientExtensions]::CreateCloudBlobClient($storageAccount)

$cacheControlValue = "public, max-age=604800"

echo "Setting cache control: $cacheControlValue"

Get-Content "imagelist.txt" | foreach {     
    $blobName = "$blobContainerName/$_".Trim()
    echo $blobName
    $blob = $blobClient.GetBlobReference($blobName)
    $blob.Properties.CacheControl = $cacheControlValue
    $blob.SetProperties()
}

Upvotes: 1

mistika
mistika

Reputation: 2491

Latest CloudBerry Explorer now supports Cache-Control: http://www.cloudberrylab.com/forum/default.aspx?g=posts&t=3047

Upvotes: 2

TMC
TMC

Reputation: 8154

The latest version of Cerebrata Cloud Storage Studio, v2011.04.23.00, supports setting cache-control on individual blob objects. Right click on the blob object, choose "View/Edit Blob Properties" then set the value for the Cache-Control attribute. (e.g. public, max-age=2592000).

If you check the HTTP headers of the blob object using curl, you'll see the cache-control header returned with the value you set.

Upvotes: 6

Joel Fillmore
Joel Fillmore

Reputation: 6038

I had to run a batch job on about 600k blobs and found 2 things that really helped:

  1. Running the operation from a worker role in the same data center. The speed between Azure services is great as long as they are in the same affinity group. Plus there are no data transfer costs.

  2. Running the operation in parallel. The Task Parallel Library (TPL) in .net v4 makes this really easy. Here is the code to set the cache-control header for every blob in a container in parallel:

    // get the info for every blob in the container
var blobInfos = cloudBlobContainer.ListBlobs(
    new BlobRequestOptions() { UseFlatBlobListing = true });
Parallel.ForEach(blobInfos, (blobInfo) =>
{
    // get the blob properties
    CloudBlob blob = container.GetBlobReference(blobInfo.Uri.ToString());
    blob.FetchAttributes();

    // set cache-control header if necessary
    if (blob.Properties.CacheControl != YOUR_CACHE_CONTROL_HEADER)
    {
        blob.Properties.CacheControl = YOUR_CACHE_CONTROL_HEADER;
        blob.SetProperties();
    }
});

Upvotes: 26

Related Questions