CODEWITHSUNDEEP
pythonuploadamazon-s3python-requests
Simon Steinberger
Simon Steinberger

Reputation: 6825

Upload images and metadata to public Amazon S3 bucket with Python requests

I know, there's the boto library for Python, however, all I'd like to do is uploading a lot of image files including metadata to a public S3 bucket. The images should go into various sub-directories inside the bucket.

With cURL, this is supposed to be working:

curl -v -F "key=test/test.jpg" -F "[email protected]" http://my-public-bucket.s3.amazonaws.com/

So I figure that should be doable with urllib, urllib2 and/or Python requests only. But how? I'm totally new to Amazon S3 ... and cURL.

Also what's the best way for storing some meta data along with the images? An additional JSON-string file?

Upvotes: 1

Views: 3712

Answers (4)

TheHandofTheKing
TheHandofTheKing

Reputation: 514

To upload to a signed url and requests I had to do this:

with open('photo_1.jpg', 'rb') as content_file:
    content = content_file.read()
result = requests.put(url=upload_url, headers={}, data=content)

This is bad because it loads everything into memory, but it should get you past the initial hump.

Also when using curl I had to use the a different option:

curl -X PUT --upload-file photo_1.jpg <url>

Note: When I created the url at my server with boto I set headers=None so that headers would not be an issue.

Upvotes: 0

Lukasa
Lukasa

Reputation: 15518

Your cURL string translates into roughly the following:

import requests

url = 'http://my-public-bucket.s3.amazonaws.com/'
files = {
    'key': ('', 'test/test.jpg'),
    'file': open('test.jpg', 'rb'),
}

r = requests.post(url, files=files)

The general form of Requests' multipart upload syntax is found in this StackOverflow answer.

Upvotes: 2

Simon Steinberger
Simon Steinberger

Reputation: 6825

Works with Python Requests only:

import requests
r = requests.post('my_public_bucket', files={'file': open('/path/test.txt', 'rb')}, data={'key': 'test/test.txt'})

Upvotes: 1

Alfe
Alfe

Reputation: 59426

Using boto (version 2.6.0) you'd do it like this:

import boto

connection = boto.connect_s3()
bucket = connection.get_bucket('mybucket')
key = bucket.new_key('myimage.jpg')
key.set_contents_from_filename('myimage.jpg')
key.set_metadata(...)

Make sure you've got the credentials in the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

That's it.

Upvotes: 1

Related Questions