CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3botoboto3
PhilBot
PhilBot

Reputation: 50

Python Boto3 AWS Multipart Upload Syntax

I am successfully authenticating with AWS and using the 'put_object' method on the Bucket object to upload a file. Now I want to use the multipart API to accomplish this for large files. I found the accepted answer in this question: How to save S3 object to a file using boto3

But when trying to implement I am getting "unknown method" errors. What am I doing wrong? My code is below. Thanks!

## Get an AWS Session
self.awsSession = Session(aws_access_key_id=accessKey,
aws_secret_access_key=secretKey,
aws_session_token=session_token,
region_name=region_type)  

 ...          

# Upload the file to S3
s3 = self.awsSession.resource('s3')
s3.Bucket('prodbucket').put_object(Key=fileToUpload, Body=data) # WORKS
#s3.Bucket('prodbucket').upload_file(dataFileName, 'prodbucket', fileToUpload) # DOESNT WORK
#s3.upload_file(dataFileName, 'prodbucket', fileToUpload) # DOESNT WORK

Upvotes: 2

Views: 3139

Answers (2)

Zile Rehman
Zile Rehman

Reputation: 51

Libcloud S3 wrapper transparently handles all the splitting and uploading of the parts for you.

Use upload_object_via_stream method to do so:

from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver

# Path to a very large file you want to upload
FILE_PATH = '/home/user/myfile.tar.gz'

cls = get_driver(Provider.S3)
driver = cls('api key', 'api secret key')

container = driver.get_container(container_name='my-backups-12345')

# This method blocks until all the parts have been uploaded.
extra = {'content_type': 'application/octet-stream'}

with open(FILE_PATH, 'rb') as iterator:
    obj = driver.upload_object_via_stream(iterator=iterator,
                                          container=container,
                                          object_name='backup.tar.gz',
                                          extra=extra)

For official documentation on S3 Multipart feature, refer to AWS Official Blog.

Upvotes: 0

jamesls
jamesls

Reputation: 5608

The upload_file method has not been ported over to the bucket resource yet. For now you'll need to use the client object directly to do this:

client = self.awsSession.client('s3')
client.upload_file(...)

Upvotes: 2

Related Questions