CODEWITHSUNDEEP
python-2.7google-app-engineunicode
LA_
LA_

Reputation: 20429

How to pass unicode values as urlfetch payload?

Here is the simplified code I have:

#coding=utf-8
...

def api_call(method, token, params=[], payload=None):
    ...
    headers = {
        'Content-Type': 'application/json; charset=utf-8'
    }
    try:
        response = urlfetch.Fetch(url, headers=headers, method=urlfetch.POST, payload=payload, deadline=60)
    ...

payload = '{"search":"%s"}' % ('тест') # UNICODE HERE!

result = api_call(method=method, token=MY_TOKEN, payload=payload)

This code works well with non-unicode characters, but when unicode is used, I get:

  File "/Users/me/Documents/Dev/GAE/app.py", line 38, in api_call
    response = urlfetch.Fetch(url, headers=headers, method=urlfetch.POST, payload=payload, deadline=60)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/urlfetch.py", line 271, in fetch
    return rpc.get_result()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/urlfetch.py", line 378, in _get_fetch_result
    rpc.check_success()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
    self.__rpc.CheckSuccess()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
    self.request, self.response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 201, in MakeSyncCall
    self._MakeRealSyncCall(service, call, request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 219, in _MakeRealSyncCall
    request_pb.set_request(request.Encode())
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/net/proto/ProtocolBuffer.py", line 103, in Encode
    self.Output(e)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/net/proto/ProtocolBuffer.py", line 347, in Output
    self.OutputUnchecked(e)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/urlfetch_service_pb.py", line 478, in OutputUnchecked
    out.putPrefixedString(self.payload_)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/net/proto/ProtocolBuffer.py", line 607, in putPrefixedString
    v = str(v)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 92-93: ordinal not in range(128)

What is wrong here?

Upvotes: 0

Views: 564

Answers (1)

bobince
bobince

Reputation: 536785

payload = '{"search":"%s"}' % ('тест') # UNICODE HERE!

This isn't a good way to create a JSON serialised object. Apart from the character encoding problem, you've also got issues if the templated string contains characters special in string literals, like quotes or backslashes. Suggest using the json module to create the serialised form and take care of encoding at the same time:

search = u'тест'
payload = json.dumps({'search': search})

'Content-Type': 'application/json; charset=utf-8'

There is no charset parameter for the application/json media type; this does nothing. JSON is always UTF, and by default UTF-8, anyway.

Upvotes: 3

Related Questions