CODEWITHSUNDEEP
amazon-web-servicesazureamazon-s3azure-storage-accountazcopy
Shoaib Maroof
Shoaib Maroof

Reputation: 369

AzCopy S3 to ADLS Gen2 Transfer Failing with PUT Permission Error Despite Full Access

I'm encountering an issue with AzCopy when attempting to transfer data from Amazon S3 to Azure Data Lake Storage Gen2. The error message indicates that I lack PUT permission, even though I’ve verified that I have the necessary permissions in both AWS and ADLS. I created a SAS token with all possible permissions checked, and my AWS IAM policy (shown below) should also be sufficient. I’ve also set the AWS environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) in my command prompt:

set AWS_ACCESS_KEY_ID=123
set AWS_SECRET_ACCESS_KEY=ABC

Despite these measures, the error persists. Any guidance or troubleshooting steps would be greatly appreciated.

IAM policy

{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "s3:ListBucket",                "s3:GetBucketLocation"            ],            "Resource": "arn:aws:s3:::s3_bucket"        },        {            "Effect": "Allow",            "Action": [                "s3:GetObject"            ],            "Resource": "arn:aws:s3:::s3_bucket/dir/profiles/*"        }    ]}

Azcopy command

C:\Users\[user]>azcopy copy "https://s3.eu-west-1.amazonaws.com/s3_bucket/dir/dir/dir/dir/" "https://[stg_container_name].blob.core.windows.net/raw-test?[SASTOKEN]" --recursive=true

INFO: Scanning...

INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists.

INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support



Job 67f9fe5e-f142-bf40-656c-ae0bf93f5e15 has started

Log file is located at: C:\Users\[user]\.azcopy\67f9fe5e-f142-bf40-656c-ae0bf93f5e15.log



**INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission PUT https://[stg_container_name].blob.core.windows.net/raw-test/2024/profiledata_GB_2024_06.csv.gz**

--------------------------------------------------------------------------------

RESPONSE 403: 403 Forbidden

ERROR CODE: CannotVerifyCopySource

--------------------------------------------------------------------------------

<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Forbidden

RequestId:5ae5e3b8-401e-0063-7b44-63ea97000000

Time:2025-01-10T09:46:44.2555118Z</Message></Error>

--------------------------------------------------------------------------------



panic: close of nil channel



goroutine 176 [running]:

github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).reportJobPartDoneHandler(0xc0005fe008)

        D:/a/1/s/ste/mgr-JobMgr.go:718 +0x327

created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr in goroutine 1

        D:/a/1/s/ste/mgr-JobMgr.go:203 +0xc88

this is the main error in the log file.

RESPONSE Status: 403 Forbidden Content-Length: 200 Content-Type: application/xml Date: Fri, 10 Jan 2025 10:22:33 GMT Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 X-Ms-Client-Request-Id: 57a6f2a5-919c-4084-6883-07a038e8a531 X-Ms-Error-Code: CannotVerifyCopySource X-Ms-Request-Id: 91cc4b22-801e-0042-5149-63e5cf000000 X-Ms-Version: 2023-08-03 Response Details: CannotVerifyCopySourceForbidden

2025/01/10 10:22:33 ERR: [P#0-T#2] COPYFAILED: https://s3.eu-west-1.amazonaws.com/s3_bucket/dir/dir/dir/2024/profiledata_GB_2024_31.csv.gz : 403 : 403 Forbidden. When Staging block from URL. X-Ms-Request-Id: 91cc4b22-801e-0042-5149-63e5cf000000

Upvotes: 0

Views: 48

Answers (1)

Shoaib Maroof
Shoaib Maroof

Reputation: 369

The issue was that the AWS IAM policy didn't have the "kms:Decrypt" permission for the KMS Encryption key associated with the S3 bucket

Upvotes: 0

Related Questions