CODEWITHSUNDEEP
pythonamazon-s3zipurllib2python-unicode
SudoKill
SudoKill

Reputation: 165

Uploading a zip file directly to AWS S3 using Python urllib2

I am trying to upload a zip file directly to S3 using a Python script, but running into some Unicode Decode Errors.

What I do is generate a Pre-Signed S3 Link and then upload data to it. I know the link works fine because the upload works when I use curl to do it like this:

curl -v -H "Content-Type: application/zip" -T /Path/To/Local/File.zip https://MySignedAWSS3Link

However, when I attempt this in Python using the code below, I get an error.

infile2 = open('/Path/To/Local/File.zip', 'rb')
filedata2 = infile2.read()
request2 = urllib2.Request("https://MySignedAWSS3Link",data=filedata2)
request2.add_header('Content-Type', 'application/zip')
request2.get_method = lambda: 'PUT'
url2 = opener.open(request2)

I get the following error/traceback in Python:

> Traceback (most recent call last):
  File "putFiles.py", line 44, in <module>
    url2 = opener.open(request2)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1181, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 827, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 10: ordinal not in range(128)

What am I doing wrong here?

Upvotes: 1

Views: 3286

Answers (1)

Dmitry Nedbaylo
Dmitry Nedbaylo

Reputation: 2344

According to your traceback, some string added to your request is unicode. However, according to your script example all strings are ascii encoded (since you are using Python2.7).

Weird things happen on your end, unless you set Python default encoding to UTF-8 (or OS sets it for you).

I hope, this should work for you (convert all strings into ascii):

infile2 = open('/Path/To/Local/File.zip', 'rb')
filedata2 = infile2.read()
request2 = urllib2.Request("https://MySignedAWSS3Link".encode('utf-8'),data=filedata2)
request2.add_header(str('Content-Type'), str('application/zip'))
request2.get_method = lambda: str('PUT')
url2 = opener.open(request2)

Update: This q/a might help you too

Upvotes: 1

Related Questions