CODEWITHSUNDEEP
pythonamazon-s3
susanna
susanna

Reputation: 1521

How can I upload a file to S3 without creating a temporary local file?

Is there any feasible way to upload a file which is generated dynamically to amazon s3 directly without first create a local file and then upload to the s3 server? I use Python.

Upvotes: 42

Views: 66874

Answers (11)

Suriyan Suresh
Suriyan Suresh

Reputation: 3024

In boto3, there is a simple way to upload a file content, without creating a local file using following code. I have modified JimJty example code for boto3

import boto3
from botocore.retries import bucket
import requests
from io import BytesIO
# set the values
aws_access_key_id=""
aws_secret_access_key=""
region_name=""
bucket=""
key=""

session = boto3.session.Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key, region_name=region_name)
s3_client = session.client('s3')
#download the file
url = "http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r = requests.get(url)
if r.status_code == 200:    
    #convert content to bytes, since upload_fileobj requires file like obj
    bytesIO = BytesIO(bytes(r.content))    
    with bytesIO as data:
        s3_client.upload_fileobj(data, bucket, key)

Upvotes: 3

Darshan Patel
Darshan Patel

Reputation: 1

This implementation is an example of uploading a list of images (NumPy list, OpenCV image objects) directly to S3

Note: you need to convert image objects to bytes or buffer to bytes while uploading the file that's how you can upload files without corruption error

#Consider you have images in the form of a list i.e. img_array
import boto3

s3 = boto3.client('s3')
res_url = []

for i,img in enumerate(img_array):
        s3_key = "fileName_on_s3.png"
        response = s3.put_object(Body=img.tobytes(), Bucket='bucket_name',Key=s3_key,ACL='public-read',ContentType= 'image/png')
        s3_url = 'https://bucket_name.s3.ap-south-1.amazonaws.com/'+s3_key
        res_url.append(s3_url)
#res_url is the list of URLs returned from S3 Upload

Upvotes: 0

stoicalpirate
stoicalpirate

Reputation: 358

Update for boto3:

aws_session = boto3.Session('my_access_key_id', 'my_secret_access_key')
s3 = aws_session.resource('s3')
s3.Bucket('my_bucket').put_object(Key='file_name.txt', Body=my_file)

Upvotes: -1

Bart Schelfhout
Bart Schelfhout

Reputation: 1

Given that encryption at rest is a much desired data standard now, smart_open does not support this afaik

Upvotes: -1

Harshad Vyawahare
Harshad Vyawahare

Reputation: 608

I had a dict object which I wanted to store as a json file on S3, without creating a local file. The below code worked for me:

from smart_open import smart_open

with smart_open('s3://access-key:secret-key@bucket-name/file.json', 'wb') as fout:
    fout.write(json.dumps(dict_object).encode('utf8'))

Upvotes: 2

dd.
dd.

Reputation: 460

You can try using smart_open (https://pypi.org/project/smart_open/). I used it exactly for that: writing files directly in S3.

Upvotes: 0

Naveen Agarwal
Naveen Agarwal

Reputation: 1036

def upload_to_s3(url, **kwargs):
    '''
    :param url: url of image which have to upload or resize to upload
    :return: url of image stored on aws s3 bucket
    '''

    r = requests.get(url)
    if r.status_code == 200:
        # credentials stored in settings AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
        conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, host=AWS_HOST)

        # Connect to bucket and create key
        b = conn.get_bucket(AWS_Bucket_Name)
        k = b.new_key("{folder_name}/{filename}".format(**kwargs))

        k.set_contents_from_string(r.content, replace=True,
                                   headers={'Content-Type': 'application/%s' % (FILE_FORMAT)},
                                   policy='authenticated-read',
                                   reduced_redundancy=True)

        # TODO Change AWS_EXPIRY
        return k.generate_url(expires_in=AWS_EXPIRY, force_http=True)

Upvotes: 2

JimJty
JimJty

Reputation: 1279

Here is an example downloading an image (using requests library) and uploading it to s3, without writing to a local file:

import boto
from boto.s3.key import Key
import requests

#setup the bucket
c = boto.connect_s3(your_s3_key, your_s3_key_secret)
b = c.get_bucket(bucket, validate=False)

#download the file
url = "http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r = requests.get(url)
if r.status_code == 200:
    #upload the file
    k = Key(b)
    k.key = "image1.png"
    k.content_type = r.headers['content-type']
    k.set_contents_from_string(r.content)

Upvotes: 27

Roy Hyunjin Han
Roy Hyunjin Han

Reputation: 5083

You could use BytesIO from the Python standard library.

from io import BytesIO
bytesIO = BytesIO()
bytesIO.write('whee')
bytesIO.seek(0)
s3_file.set_contents_from_file(bytesIO)

Upvotes: 19

kindall
kindall

Reputation: 184270

I assume you're using boto. boto's Bucket.set_contents_from_file() will accept a StringIO object, and any code you have written to write data to a file should be easily adaptable to write to a StringIO object. Or if you generate a string, you can use set_contents_from_string().

Upvotes: 2

jterrace
jterrace

Reputation: 67093

The boto library's Key object has several methods you might be interested in:

For an example of using set_contents_from_string, see Storing Data section of the boto documentation, pasted here for completeness:

>>> from boto.s3.key import Key
>>> k = Key(bucket)
>>> k.key = 'foobar'
>>> k.set_contents_from_string('This is a test of S3')

Upvotes: 13

Related Questions