CODEWITHSUNDEEP
pythonunicodeencodingutf-8elementtree
pistacchio
pistacchio

Reputation: 58873

ElementTree and unicode

I have this char in an xml file:

<data>
  <products>
      <color>fumè</color>
  </product>
</data>

I try to generate an instance of ElementTree with the following code:

string_data = open('file.xml')
x = ElementTree.fromstring(unicode(string_data.encode('utf-8')))

and I get the following error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 185: ordinal not in range(128)

(NOTE: The position is not exact, I sampled the xml from a larger one).

How to solve it? Thanks

Upvotes: 21

Views: 38820

Answers (6)

Martijn Pieters
Martijn Pieters

Reputation: 1121834

You do not need to decode XML for ElementTree to work. XML carries it's own encoding information (defaulting to UTF-8) and ElementTree does the work for you, outputting unicode:

>>> data = '''\
... <data>
...   <products>
...       <color>fumè</color>
...   </products>
... </data>
... '''
>>> x = ElementTree.fromstring(data)
>>> x[0][0].text
u'fum\xe8'

If your data is contained in a file(like) object, just pass the filename or file object directly to the ElementTree.parse() function:

x = ElementTree.parse('file.xml')

Upvotes: 12

gitaarik
gitaarik

Reputation: 46300

Might you have stumbled upon this problem while using Requests (HTTP for Humans), response.text decodes the response by default, you can use response.content to get the undecoded data, so ElementTree can decode it itself. Just remember to use the correct encoding.

More info: http://docs.python-requests.org/en/latest/user/quickstart/#response-content

Upvotes: 35

Lukas Graf
Lukas Graf

Reputation: 32590

You need to decode utf-8 strings into a unicode object. So

string_data.encode('utf-8')

should be

string_data.decode('utf-8')

assuming string_data is actually an utf-8 string.

So to summarize: To get an utf-8 string from a unicode object you encode the unicode (using the utf-8 encoding), and to turn a string to a unicode object you decode the string using the respective encoding.

For more details on the concepts I suggest reading The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (not Python specific).

Upvotes: 16

hejhula
hejhula

Reputation: 11

Function open() does not return a string. Instead use open('file.xml').read().

Upvotes: 1

Jon Clements
Jon Clements

Reputation: 142146

Have you tried using the parse function, instead of opening the file... (which BTW would require a .read() after it for the .fromstring() to work...)

import xml.etree.ElementTree as ET

tree = ET.parse('file.xml')
root = tree.getroot()
# etc...

Upvotes: 2

Maksym Polshcha
Maksym Polshcha

Reputation: 18358

The most likely your file is not UTF-8. è character can be from some other encoding, latin-1 for example.

Upvotes: 1

Related Questions