Python generate_blob_sas doesn't create right SAS token (failed on copy operation in azcopy)

Question

It's my first question at such cool resource.

Description of a goal: I need to copy from one Azure storage account to another. ... Not only this, but this part doesn't work for me now.

Problem: If generate token with generate_blob_sas you will have in azcopy operation:

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.13.1-0.20210914164749-2d6cd3e07548/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) ===== Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

If insert manually created token on Azure portal - works fine.

Why I use azcopy inside of Python, because I have structure like this:

.
|
|└── Folder1
|    ├── file1.json
|    ├── file2.json
|    ├── file3.json
|    ├── file4.json
|    ├── file5.son
|    └── Folder2
|        ├── fileswithsomenames1.html
|        ├── fileswithsomenames2.html
|        ├── fileswithsomenames3.html
|        ├── fileswithsomenames4.html
|        ├── fileswithsomenames5.html
|        ├── fileswithsomenames6.html
|        ├── fileswithsomenames7.html
|        ├── fileswithsomenames8.html
|        ├── fileswithsomenames9.html
|        └── fileswithsomenames10.html
└──receipt.json (this file will be eachtime but with different content)

A number of files will be different in each situation. And I need to work ahead with files from target blob. I didn't find understandable description how to copy from store1/conteiner1/{var.name}-blob to var.name-store2/conteainer1/ all content including many folders. Almost all descriptions and manuals for 1 blob file.

My code:

code to authorise and get storage account name and string 
...
customer_name = 'abc-qa'
...
container_name_source = "container1"
blob_name_source = customer_name+"-blobfolder1/blobfolder2"

container_name_target = customer_name+"-bl"
blob_name_target = "/"

# Function to generate sas for source blob
def get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source):
    sas_blob_source = generate_blob_sas(account_name=account_name_source, 
                                container_name=container_name_source,
                                blob_name=blob_name_source,
                                account_key=account_key_1_source,
                                permission=BlobSasPermissions(read=True),
                                expiry=datetime.utcnow() + timedelta(hours=1))
    return sas_blob_source

sas_blob_source = get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source)
url_blob_source = 'https://'+account_name_source+'.blob.core.windows.net/'+container_name_source+'/'+blob_name_source+'?'+sas_blob_source

# Function to generate sas for target blob
def get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target):
    sas_blob_target = generate_blob_sas(account_name=account_name_target, 
                                container_name=container_name_target,
                                blob_name=blob_name_target,
                                account_key=account_key_1_target,
                                permission=BlobSasPermissions(add=True, create=True, write=True, tag=False, delete_previous_version=True),
                                expiry=datetime.utcnow() + timedelta(hours=1))
    return sas_blob_target

sas_blob_target = get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target)
print(sas_blob_target)
url_blob_target = 'https://'+account_name_target+'.blob.core.windows.net/'+container_name_target+'/?'+sas_blob_target




#copy seed-data folder from account_name_source/container1/customer_name+"-blobfolder1/blobfolder2" to account_name_target/customer_name+"-bl"
cmd = f"azcopy copy '{url_blob_source}' '{url_blob_target}' --recursive=True"
print(str(cmd))
results = subprocess.run(
    str(cmd), shell=True, universal_newlines=True, check=True)
print(results.stdout)

Result if gen manually 'sp=rl&st=2022-04-14T14:29:48Z&se=2022-04-14T22:29:48Z&spr=https&sv=2020-08-04&sr=c&sig={token}'

Rusult if gen over generate_blob_sas

'se=2022-04-14T17%3A18%3A48Z&sp=r&sv=2020-10-02&sr=b&sig={token}'

Answer 1

This is a working code for me now with generate_container_sas from access policy:

# Here code where I get storage account name and key. And Authentication by SPN which in KV.  
...
customer_name = 'abc-qa'
container_name_source = "conteiner1"
blob_name_source = customer_name+"-blobfolder1/blobfolder2"

container_name_target = customer_name+"-bl"
blob_name_target = "blobfolder2"


from datetime import datetime, timedelta

connection_string_source = 'DefaultEndpointsProtocol=https;AccountName='+account_name_source+';AccountKey='+blob_key_1_source+';EndpointSuffix=core.windows.net'
connection_string_target = 'DefaultEndpointsProtocol=https;AccountName='+account_name_target+';AccountKey='+blob_key_1_target+';EndpointSuffix=core.windows.net'


# Instantiate a BlobServiceClient using a connection string
from azure.storage.blob import BlobServiceClient
blob_service_client_source = BlobServiceClient.from_connection_string(connection_string_source)
blob_service_client_target = BlobServiceClient.from_connection_string(connection_string_target)

# Instantiate a ContainerClient
container_client_source = blob_service_client_source.get_container_client(container_name_source)
container_client_target = blob_service_client_target.get_container_client(container_name_target)

# [START set_container_access_policy]
# Create access policy Soure
from azure.storage.blob import AccessPolicy, ContainerSasPermissions
access_policy_source = AccessPolicy(permission=ContainerSasPermissions(read=True, list=True),
                              expiry=datetime.utcnow() + timedelta(hours=1),
                              start=datetime.utcnow() - timedelta(minutes=1))

identifiers_source = {'rl': access_policy_source}

# Create access policy Target
from azure.storage.blob import AccessPolicy, ContainerSasPermissions
access_policy_target = AccessPolicy(permission=ContainerSasPermissions(write=True),
                              expiry=datetime.utcnow() + timedelta(hours=1),
                              start=datetime.utcnow() - timedelta(minutes=1))

identifiers_target = {'w': access_policy_target}

# Set the access policy on the container
container_client_source.set_container_access_policy(signed_identifiers=identifiers_source)
container_client_target.set_container_access_policy(signed_identifiers=identifiers_target)

# [END set_container_access_policy]

# [START get_container_access_policy]
policy_source = container_client_source.get_container_access_policy()
policy_source = container_client_target.get_container_access_policy()
# [END get_container_access_policy]

# [START generate_sas_token]
# Use access policy to generate a sas token Source
from azure.storage.blob import generate_container_sas

sas_token_source = generate_container_sas(
    container_client_source.account_name,
    container_client_source.container_name,
    account_key=container_client_source.credential.account_key,
    policy_id='rl'
)
print(sas_token_source)
# [END generate_sas_token]

# Use access policy to generate a sas token Source
from azure.storage.blob import generate_container_sas

sas_token_target = generate_container_sas(
    container_client_target.account_name,
    container_client_target.container_name,
    account_key=container_client_target.credential.account_key,
    policy_id='w'
)
print(sas_token_target)
# [END generate_sas_token]

#copy seed-data folder from stcorpmainstorage/seed-data/$(name of env)-seed-data/seed-data to $(name of env)store
cmd = f"azcopy copy 'https://{account_name_source}.blob.core.windows.net/{container_name_source}/{blob_name_source}?{sas_token_source}' 'https://{account_name_target}.blob.core.windows.net/{container_name_target}/?{sas_token_target}' --recursive=True"
results = subprocess.run(
    str(cmd), shell=True, universal_newlines=True, check=True)
print(results.stdout)

...
# Here code to work with target blob and next steps.

I hope it will be useful for someone.

Answer 2

The reason why your code is failing with the SAS token generated programmatically is because it is missing List permission. If you look at the working SAS token, it has both Read and List permissions (sp=rl) whereas the non-working SAS token only has Read permission (sp=r).

Another issue is that in your code you are generating SAS token for the blob (sr=b) whereas the working SAS token is for the container (sr=c). This is for both source and target SAS token.

To fix this:

• Create the SAS token with both Read and List permissions.
• Create the SAS token for the container instead of blob. For this, you will need to use generate_container_sas.

Once you do these two things, you should not get the error.