CODEWITHSUNDEEP
pythonjinja2
shane
shane

Reputation: 1055

Python - 'ascii' codec can't decode byte

I'm using Python 2.6 and Jinja2 to create HTML reports. I provide the template with many results and the template loops through them and creates HTML tables

When calling template.render, I've suddenly started getting this error.

<td>{{result.result_str}}</td>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128)

The strange thing is, even if I set result.result_str to a simple ascii string like "abc" for every result, I am still seeing this error. I'm new to Jinja2 and Python and would appreciate any ideas on how I can go about investigating the problem to get to the root cause.

Upvotes: 41

Views: 62929

Answers (7)

Dae
Dae

Reputation: 2427

Just encountered the same problem in a piece of code which saves output from Jinja2 to HTML files:

with open(path, 'wb') as fh:
    fh.write(template.render(...))

It's easy to blame Jinja2, although the actual problem is in Python's open() which as of version 2.7 doesn't support UTF-8. The fix is as simple as:

import codecs
with codecs.open(path, 'wb', 'utf-8') as fh:
    fh.write(template.render(...))

Upvotes: 11

Martin Stone
Martin Stone

Reputation: 13007

From http://jinja.pocoo.org/docs/api/#unicode

Jinja2 is using Unicode internally which means that you have to pass Unicode objects to the render function or bytestrings that only consist of ASCII characters.

So wherever you set result.result_str, you need to make it unicode, e.g.

result.result_str = unicode(my_string_variable, "utf8")

(If your bytes were utf8 encoded unicode)

or

result.result_str = u"my string"

Upvotes: 43

Zinovy Nis
Zinovy Nis

Reputation: 463

Or you may do

export LANG='en_US.UTF-8'

in your console where you run the script.

Upvotes: -1

Richard Huang
Richard Huang

Reputation: 947

Try to add this:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

It fixed my problem, good luck.

Upvotes: 78

cat
cat

Reputation: 2895

Simple strings may contain UTF-8 character bytes but they are not of type unicode. This can be fixed by "decode" which converts str to unicode. Works in Python 2.5.5.

my_string_variable.decode("utf8")

Upvotes: 5

jd.
jd.

Reputation: 10958

If you get an error with a string like "ABC", maybe the non-ASCII character is somewhere else. In the template source perhaps?

In any case, use Unicode strings throughout your application to avoid this kind of problems. If your data source provides you with byte strings, you get unicode strings with byte_string.decode('utf-8'), if the string is encoded in UTF-8. If your source is a file, use the StreamReader class in the codecs module.

If you're unsure about the difference between Unicode strings and regular strings, read this: http://www.joelonsoftware.com/articles/Unicode.html

Upvotes: 20

tchrist
tchrist

Reputation: 80415

ASCII is a 7-bit code. The value 0xC4 cannot be stored in 7 bits. Therefore, you are using the wrong encoding for that data.

Upvotes: 0

Related Questions