How to handle this exception 'ascii' codec can't decode?

Question

I get 'ascii' codec can't decode byte 0xe2 in position 620: ordinal not in range(128) again in my log files from appengine and I don't know what to do about it. The offending code appears to be 'name': form.name.data.decode('utf-8') where I already tried everything to make it work, encode, decode and unicode() without knowing why it's not builtin. I usually get the answer here that I must know what the input is but why? And when I try with some foreign chars like åäö it does work so it's not obvious what the problem is. The full stacktrace is

'ascii' codec can't decode byte 0xe2 in position 620: ordinal not in range(128)
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/apps/s~montaoproject/2015p.373396352072543150/authhandlers.py", line 34, in dispatch
    webapp2.RequestHandler.dispatch(self)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~montaoproject/2015p.373396352072543150/main.py", line 2610, in post
    'name': form.name.data.decode('utf-8')
  File "/base/data/home/apps/s~montaoproject/2015p.373396352072543150/authhandlers.py", line 104, in render
    self.response.write(self.jinja2.render_template(template_name, **values))
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2_extras/jinja2.py", line 158, in render_template
    return self.environment.get_template(_filename).render(**context)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/jinja2-2.6/jinja2/environment.py", line 894, in render
    return self.environment.handle_exception(exc_info, True)
  File "templates/2.html", line 1680, in top-level template code
    <div class="table_data"> {{form.text(cols="45", rows="10", placeholder=_('Please describe the product or service in a few simple sentences'))|safe}}{% if form.text.errors %} <div class="maintext">
  File "/base/data/home/apps/s~montaoproject/2015p.373396352072543150/wtforms/fields/core.py", line 139, in __call__
    return self.widget(self, **kwargs)
  File "/base/data/home/apps/s~montaoproject/2015p.373396352072543150/main.py", line 2034, in __call__
    return super(MyTextArea, self).__call__(field, **kwargs)
  File "/base/data/home/apps/s~montaoproject/2015p.373396352072543150/wtforms/widgets/core.py", line 221, in __call__
    return HTMLString('<textarea %s>%s</textarea>' % (html_params(name=field.name, **kwargs), escape(text_type(field._value()))))
  File "/base/data/home/apps/s~montaoproject/2015p.373396352072543150/wtforms/fields/core.py", line 496, in _value
    return text_type(self.data) if self.data is not None else ''
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 620: ordinal not in range(128)

Update 140209 06:02 CET

I also get the error using decode

'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/apps/s~montaoproject/2015r.373623653798528527/authhandlers.py", line 34, in dispatch
    webapp2.RequestHandler.dispatch(self)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~montaoproject/2015r.373623653798528527/main.py", line 2664, in post
    'name': form.name.data.encode('utf-8')
  File "/base/data/home/apps/s~montaoproject/2015r.373623653798528527/authhandlers.py", line 104, in render
    self.response.write(self.jinja2.render_template(template_name, **values))
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2_extras/jinja2.py", line 158, in render_template
    return self.environment.get_template(_filename).render(**context)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/jinja2-2.6/jinja2/environment.py", line 894, in render
    return self.environment.handle_exception(exc_info, True)
  File "templates/2.html", line 223, in top-level template code
    {{ form.phonenumber(size="17", maxlength="14")|safe }}<label for="phone_hidden">{{ form.phoneview|safe }}{% trans %}View on site{% endtrans %}</label><br />{% trans %}Use areacode{% endtrans %}<br>
  File "/base/data/home/apps/s~montaoproject/2015r.373623653798528527/wtforms/fields/core.py", line 139, in __call__
    return self.widget(self, **kwargs)
  File "/base/data/home/apps/s~montaoproject/2015r.373623653798528527/wtforms/widgets/core.py", line 123, in __call__
    kwargs['value'] = field._value()
  File "/base/data/home/apps/s~montaoproject/2015r.373623653798528527/wtforms/fields/core.py", line 496, in _value
    return text_type(self.data) if self.data is not None else ''
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)

Answer 1

You should use encode rather than decode. What you have in name.data is a unicode string (you can understand that from the byte 0xe2 in error message) and you need to transform that into bytes string.

I've done some ipython fiddling:

In [32]: unichr(0xe2)
Out[32]: u'\xe2'

In [33]: unichr(0xe2).encode("utf-8")
Out[33]: '\xc3\xa2'

In [34]: print unichr(0xe2).encode("utf-8")
â