CODEWITHSUNDEEP
pythonamazon-web-servicesaws-lambdaboto3
vanquish
vanquish

Reputation: 135

boto3 s3 copyObject error

I'm trying to copy a file from 1 bucket to another prefix in the same bucket using lambda & boto3 however I keep getting an error:

An error occurred (AccessDenied) when calling the CopyObject operation.

or

An error occurred (403) when calling the HeadObject operation: Forbidden

depending on which copy method I use.

The lambda function has a role assigned to it, which I think gives it all the permissions it requires:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "s3:HeadObject",
            "s3:ListObjects"
        ],
        "Resource": [
            "arn:aws:s3:::bucket-name",
            "arn:aws:s3:::bucket-name/*"
        ],
        "Effect": "Allow"
    },
    {
        "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject"
        ],
        "Resource": [
            "arn:aws:s3:::bucket-name/folderA/folderB/*",
            "arn:aws:s3:::bucket-name/folderC/folderD/*",
            "arn:aws:s3:::bucket-name/folderE/folderF/*"
        ],
        "Effect": "Allow"
    }
]
}

The lambda function is:

#connect to s3
s3 = boto3.resource('s3')

dirs = {
    "folderA/folderB": "folderC/folderD"        
}    

key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
etag = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['eTag'], encoding='utf-8')    
bucket = event['Records'][0]['s3']['bucket']['name']

filePathName = key.split("/")
sourceDir = filePathName[0] + "/" + filePathName[1]
fileName = filePathName[2]

sourceKey = sourceDir + "/" + fileName
source = {'Bucket': bucket, 'Key': sourceKey}
destination = dirs[sourceDir] + "/" + fileName

##########
# This option comes up with the An error occurred (AccessDenied) when calling the CopyObject operation. Error
###########
s3.Object(bucket, destination).copy_from(CopySource=source)

###########
## This option comes up with the An error occurred (403) when calling the HeadObject operation: Forbidden error
###########
s3.meta.client.copy(source, bucket, destination)

EDIT: forgot to mention, it works fine if I change the role to 

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::bucket-name",
            "arn:aws:s3:::bucket-name/*"
        ],
        "Effect": "Allow"
    }

Upvotes: 5

Views: 12744

Answers (3)

awsconfusion
awsconfusion

Reputation: 1

I noticed that many of the other solutions here are not particularly "turn-key", especially in terms of handling large buckets or working in an environment where aws-cli is not available.

This one doesn't rely on aws-cli and uses the boto3 paginators to handle large buckets.

import boto3

def recursive_s3_cp(source_bucket, source_bucket_path, dest_bucket, dest_bucket_path):
    s3_client = boto3.client("s3")
    paginator = s3_client.get_paginator('list_objects_v2')
    response_iterator = paginator.paginate(
            Bucket=source_bucket,
            Prefix=source_bucket_path,
            Delimiter="/"
            )
    result = response_iterator.build_full_result()
    # Skip first item as it's a reference to the prefix directory
    for bucket_object in result['Contents'][1:]:
        new_path = bucket_object['Key'].replace(source_bucket_path, dest_bucket_path)
        response = s3_client.copy_object(
                # Tons of other parameters you can pass here if you need them
                Bucket=dest_bucket,
                CopySource={
                    'Bucket': source_bucket,
                    'Key': bucket_object['Key']
                    },
                Key=new_path,
                )
        if not response['ResponseMetadata']['HTTPStatusCode'] == 200:
            raise Exception('s3 copy failed with response: ', response)

Upvotes: 0

Vevs
Vevs

Reputation: 47

This question is old, but maybe for someone will be helpful:

copy_source = {
    'Bucket': 'my_perfect_bucket',
    'Key': 'media/old_path/glory.jpg'
}

s3.Object('my_perfect_bucket','media/new_path/glory.jpg').copy_from(CopySource=copy_source)

Upvotes: 1

bluppfisk
bluppfisk

Reputation: 2652

I was running into a similar problem. The solution: source in CopySource=source had to be a full path from the bucket root to the actual file instead of a dictionary of bucket name and key. So I think your code may have to be:

s3.Object(bucket, destination).copy_from(CopySource=bucket + sourceDir)

Upvotes: 10

Related Questions