CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3boto3
gauss02
gauss02

Reputation: 1

creating-large-zip-files-in-aws-s3 chunks and back to S3 python

is it possible to use upload_fileobj() with python generator like:

def example():
 gen = (range(3))
 for i in gen:
   yield BytesIO(i)

then, by using boto3 we must write in the same file (the same key file)

for i in example():
  client.upload_fileobj(i)

Thanks a lot for your response

Upvotes: 0

Views: 589

Answers (1)

ogdenkev
ogdenkev

Reputation: 2374

It sounds like you are trying to save all the bytes yielded by the generator to the same object on S3. If so, you can do that with a multipart upload. However, each part must be at least 5 MB (which presumably your generator would yield 5 MB chunks).

import boto3

client = boto3.client("s3")
my_bucket = "examplebucket"
my_key = "largeobject"

# Initiate multipart upload
response = client.create_multipart_upload(
    Bucket=my_bucket,
    Key=my_key,
)
# Record the UploadId
upload_id = response["UploadId"]

# Upload parts and keep track of them for completion
parts = []
for i, chunk in enumerate(example()):
    response = client.upload_part(
        Body=chunk,
        Bucket=my_bucket,
        Key=my_key,
        PartNumber=i,
        UploadId=upload_id,
    )
    parts.append({
        "ETag": response["ETag"],
        "PartNumber": i
    })

# Complete the multipart upload
client.complete_multipart_upload(
    Bucket=my_bucket,
    Key=my_Key,
    MultipartUpload={
        'Parts': parts
    },
    UploadId=upload_id
)

Upvotes: 1

Related Questions