CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3boto
markwalker_
markwalker_

Reputation: 12849

Connection closed error from Amazon S3 after changing to boto get_bucket

Yesterday I started using Amazon S3 with boto, I setup a python script to create a bucket and upload images in a directory. I ran it several times and it seemed to be okay.

I don't want to constantly be creating buckets however so I adapted my script to get_bucket:

import os
import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket('images')

# Run Selenium test process to create images


for root, dirs, files in os.walk(imagesPath):
for name in files:       
   filename = os.path.join(root, name)
   key = bucket.new_key('{0}/{1}/{2}'.format(revisionNumber, images_process, name)) 
   print "Uploading " + filename + " to Amazon S3"
   key.set_contents_from_filename(filename)
   key.set_acl('public-read')

The error comes from the line key.set_contents_from_filename(filename)

This code is essentially boto sample code from the S3 getting started guide which I've put into a loop. I assume the problem is that its great to do things the first time, but isn't robust enough to cope with overwriting etc. Do I have to give boto more than just the filename to upload?

The output from the script is:

 [exec] Uploading images/Add_Employer_Process/20227M/1.png to Amazon S3
 [exec] Traceback (most recent call last):
 [exec]   File "addEmployerProcess.py", line 121, in <module>
 [exec]     k.set_contents_from_filename(filename)
 [exec]   File "C:\Python26\lib\site-packages\boto\s3\key.py", line 969, inset_contents_from_filename
 [exec]     encrypt_key=encrypt_key)
 [exec]   File "C:\Python26\lib\site-packages\boto\s3\key.py", line 902, inset_contents_from_file
 [exec]     size=size)
 [exec]   File "C:\Python26\lib\site-packages\boto\s3\key.py", line 660, insend_file
 [exec]     query_args=query_args)
 [exec]   File "C:\Python26\lib\site-packages\boto\s3\connection.py", line 449, in make_request
 [exec]     override_num_retries=override_num_retries)
 [exec]   File "C:\Python26\lib\site-packages\boto\connection.py", line 829, in make_request
 [exec]     return self._mexe(http_request, sender, override_num_retries)
 [exec]   File "C:\Python26\lib\site-packages\boto\connection.py", line 794, in _mexe
 [exec]     raise e
 [exec] socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host

I'm new to all of this, python included, so any help would be appreciated :)

Upvotes: 0

Views: 3728

Answers (1)

garnaat
garnaat

Reputation: 45856

Your code looks fine although one small, unrelated suggestion would be to set the ACL policy on the file at the time you are writing the file rather than as a separate step, like this:

key.set_contents_from_filename(filename, policy='public-read')

I tried your code locally and it worked just fine. I'm not sure why you are getting the socket exception but the really odd thing is that boto should be catching that exception and retrying the request automatically.

It would be useful to get some debug logging. You can enable full debug logging to the console like this:

import boto
boto.set_stream_logger('foo')
...
s3 = boto.connect_s3(debug=2)

This might provide more information about what is going on.

Upvotes: 1

Related Questions