CODEWITHSUNDEEP
pythonunicodexml-parsing
jlconlin
jlconlin

Reputation: 15084

Unicode error when passing unicode object to XML parser

I am trying to read a gzip file which contains xml and unicode, but I'm getting an error. The code I am using is:

import gzip
import xml

path = "index.mjml.gz"
gzFile = gzip.open(path, mode='r')
gzContents = gzFile.read()
gzFile.close()

unicodeContents = gzContents.encode('utf-8')
xmlContent = xml.dom.minidom.parseString(unicodeContents)
# Do stuff with xmlContent

When I run this code I get the following error (fails on the line that starts with xmlContent)

/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/xml/dom/minidom.pyc in parseString(string, parser)
   1922     if parser is None:
   1923         from xml.dom import expatbuilder
-> 1924         return expatbuilder.parseString(string)
   1925     else:
   1926         from xml.dom import pulldom

/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/xml/dom/expatbuilder.pyc in parseString(string, namespaces)
    938     else:
    939         builder = ExpatBuilder()
--> 940     return builder.parseString(string)
    941 
    942 

/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/xml/dom/expatbuilder.pyc in parseString(self, string)
    221         parser = self.getParser()
    222         try:
--> 223             parser.Parse(string, True)
    224             self._setup_subset(string)
    225         except ParseEscape:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1141336: ordinal not in range(128)

I found a previous answer similar to this Reading utf-8 characters from a gzip file in python, but I'm still getting an error.

Is there a problem with the xml parser?

(I'm using Python 2.7.?)

Upvotes: 2

Views: 5813

Answers (1)

ekhumoro
ekhumoro

Reputation: 120758

You can't pass a unicode string to xml.dom.minidom.parseString.

It has to be an appropriately encoded byte string:

>>> import xml.dom.minidom as xmldom
>>>
>>> source = u"""\
... <?xml version="1.0" encoding="utf-8"?>
... <root><text>Σὲ γνωρίζω ἀπὸ τὴν κόψη</text></root>
... """
>>> doc = xmldom.parseString(source.encode('utf-8'))
>>> print doc.getElementsByTagName('text')[0].toxml()
<text>Σὲ γνωρίζω ἀπὸ τὴν κόψη</text>

EDIT

Just to clarify - the stream read from the gzipped xml file should be passed directly to the parser without attempting to encode or decode it:

import gzip
import xml

path = "index.mjml.gz"
gzFile = gzip.open(path, mode='r')
gzContents = gzFile.read()
gzFile.close()

xmlContent = xml.dom.minidom.parseString(gzContents)

The parser will read the encoding from the xml declaration at the start of the file (or assume "utf-8" if there isn't one). It can then use this to decode the contents to unicode.

Upvotes: 5

Related Questions