CODEWITHSUNDEEP
pythonxmlutf-8elementtree
mart1n
mart1n

Reputation: 6213

ElementTree does not write data in UTF-8

I'm pulling in data from a database and attempting to create an XML file from this data. The data is in UTF-8 and can contain characters such as á, š, or č. This is the code:

import xml.etree.cElementTree as ET

tree = ET.parse(metadata_file)
# ..some commands that alter the XML..
tree.write(metadata_file, encoding="UTF-8")

When writing the data, the script fails with:

Traceback (most recent call last):
  File "get-data.py", line 306, in <module>
    main()
  File "get-data.py", line 303, in main
    tree.write(metadata_file, encoding="UTF-8")
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 820, in write
    serialize(write, self._root, encoding, qnames, namespaces)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml
    _serialize_xml(write, e, encoding, qnames, None)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml
    _serialize_xml(write, e, encoding, qnames, None)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 937, in _serialize_xml
    write(_escape_cdata(text, encoding))
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1073, in _escape_cdata
    return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)

The only way to prevent this is to decode the data written to the XML file with:

text = text.decode('utf-8')

but then the resulting file will contain e.g. &#269; rather than a č. Any idea how I can write the data to the file and keep it in UTF-8?

Edit:

This is the example of what the script does:

]$ echo "<data></data>" > test.xml
]$ cat test.xml
<data></data>
]$ python
Python 2.7.5 (default, Nov  3 2014, 14:33:39)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.etree.cElementTree as ET
>>> tree = ET.parse('./test.xml')
>>> root = tree.getroot()
>>> new = ET.Element("elem")
>>> new.text = "á, š, or č"
>>> root.append(new)
>>> tree.write('./text.xml', encoding="UTF-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 820, in write
    serialize(write, self._root, encoding, qnames, namespaces)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml
    _serialize_xml(write, e, encoding, qnames, None)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 937, in _serialize_xml
    write(_escape_cdata(text, encoding))
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1073, in _escape_cdata
    return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Upvotes: 1

Views: 5364

Answers (2)

mart1n
mart1n

Reputation: 6213

Ah, finally got it, this is the correct way to do it:

]$ echo "<data></data>" > text.xml
]$ python
Python 2.7.5 (default, Nov  3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.etree.cElementTree as ET
>>>
>>> tree = ET.parse('./test.xml')
>>> root = tree.getroot()
>>> new = ET.Element("elem")
>>> new.text = "á, š, or č".decode('utf-8')
>>> root.append(new)
>>> tree.write('./textout.xml', encoding="UTF-8")
>>>
>>> exit()
]$ cat textout.xml
<?xml version='1.0' encoding='UTF-8'?>
<data><elem>á, š, or č</elem></data>

In my original solution, I was encoding it as UTF-8 in the write() but not decoding it with .decode('utf-8').

Upvotes: 1

ekhumoro
ekhumoro

Reputation: 120598

The question does not make clear what kind of object metadata_file is.

If an ordinary file object is used, there are no errors, and the output is as expected:

>>> import xml.etree.cElementTree as ET
>>> stream = open('test.xml', 'wb+')
>>> stream.write(u"""\
... <root>characters such as á, š, or č.</root>
... """.encode('utf-8'))
>>> stream.seek(0)
>>> tree = ET.parse(stream)
>>> stream.close()
>>> ET.tostring(tree.getroot())
'<root>characters such as &#225;, &#353;, or &#269;.</root>'
>>> stream = open('test.xml', 'w')
>>> tree.write(stream, encoding='utf-8', xml_declaration=True)
>>> stream.close()
>>> open('test.xml').read()
"<?xml version='1.0' encoding='utf-8'?>\n<root>characters such as \xc3\xa1, \xc5\xa1, or \xc4\x8d.</root>"

Upvotes: 0

Related Questions