Merging XML files using Python's ElementTree and maintaining CDATA tags

Question

I pretty much reused the same bit of code from here merging xml files using python's ElementTree and I got it working. The XML files I am trying to merge look like this

A.xml

<root>
  <categories>
    <category name="Biology" />
  </categories>
  <app>
    <mainHeader><![CDATA[AP Biology]]></mainHeader>
    <questions>
      <question type="0" number="1" title="Biology #1">
        <images />
        <description><![CDATA[<b>Which of the following is 
        the site of protein synthesis?</b>]]></description>
        <category><![CDATA[Biology]]></category>
        <choices>
          <choice name="A"><![CDATA[Cell wall]]></choice>
          <choice name="B" correct_answer="true"><![CDATA[Ribosomes]]></choice>
          <choice name="C"><![CDATA[Vacuoles]]></choice>
          <choice name="D"><![CDATA[DNA polymerase]]></choice>
          <choice name="E"><![CDATA[RNA polymerase]]></choice>
        </choices>
        <explanation><![CDATA[<b>Answer:</b> B, Ribosomes.  Translation, the 
        process that converts mRNA code into protein, takes place in ribosomes.
        <br /><br /><b>Key Takeaway: </b>Ribosomes are complexes of RNA and 
        protein that are located in cell nuclei.  Ribosomes catalyze both the 
        conversion of the mRNA code into amino acids as well as the assembly of 
        the individual amino acids into a peptide change that becomes a protein.
        ]]></explanation>
      </question>
    </questions>
  </app>
</root>

B.xml

<root>
  <categories>
    <category name="Biology" />
  </categories>
  <app>
    <mainHeader><![CDATA[SAT Biology]]></mainHeader>
    <questions>
      <question type="0" number="1" title="Biology #1">
        <images>
        </images>
        <category><![CDATA[Biology]]></category>
        <description><![CDATA[<b>The site of cellular respiration 
        is:</b>]]></description>
        <choices>
          <choice name="A"><![CDATA[DNA polymerase]]></choice>
          <choice name="B"><![CDATA[Ribosomes]]></choice>
          <choice name="C" correct_answer="true"><![CDATA[Mitochondria]]></choice>
          <choice name="D"><![CDATA[RNA polymerase]]></choice>
          <choice name="E"><![CDATA[Vacuoles]]></choice>
        </choices>
        <explanation><![CDATA[<b>Answer:</b> C, Mitochondria.  
        The mitochondrion (plural mitochondria) is known as the “powerhouse” 
        of the cell for its role in energy production.<br /><br />
        <b>Key Takeaway: </b>The mitochondrion is a membrane-bound organelle 
        found in most eukaryotic cells.  The dominant role of the mitochondrion 
        is the production of ATP through cellular respiration, which is 
        dependent on the presence of oxygen.  All forms of cellular 
        respiration, glycolysis, Krebs’ cycle, and oxidative phosphorylation, 
        take place within the mitochondria.]]></explanation>
      </question>
    </questions>
  </app>
</root>

This is the code I used to merge them

import os, os.path, sys
import glob
from xml.etree import ElementTree

def run(files):
    xml_files = glob.glob(files +"/*.xml")
    xml_element_tree = None
    for xml_file in xml_files:
        data = ElementTree.parse(xml_file).getroot()
        # print ElementTree.tostring(data)
        for question in data.iter('questions'):
            if xml_element_tree is None:
                xml_element_tree = data 
                insertion_point = xml_element_tree.find('app').findall("./questions")[0]
            else:
                insertion_point.extend(question) 
    if xml_element_tree is not None:
        print ElementTree.tostring(xml_element_tree)

And it works except that the output does not maintain the CDATA tags. To be specific, this is the output I get.

<root>
  <categories>
    <category name="Biology" />
  </categories>
  <app>
    <mainHeader>AP Biology</mainHeader>
    <questions>
      <question number="1" title="Biology #1" type="0">
        <images />
        <category>Biology</category>
        <description>&lt;b&gt;Which of the following is the site 
        of protein synthesis?&lt;/b&gt;</description>
        <choices>
          <choice name="A">Cell wall</choice>
          <choice correct_answer="true" name="B">Ribosomes</choice>
          <choice name="C">Vacuoles</choice>
          <choice name="D">DNA polymerase</choice>
          <choice name="E">RNA polymerase</choice>
        </choices>
        <explanation>&lt;b&gt;Answer:&lt;/b&gt; B, Ribosomes.  
        Translation, the process that converts mRNA code into protein, 
        takes place in ribosomes.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;
        Key Takeaway: &lt;/b&gt;Ribosomes are complexes of RNA and protein 
        that are located in cell nuclei.  Ribosomes catalyze both the 
        conversion of the mRNA code into amino acids as well as the assembly 
        of the individual amino acids into a peptide change that becomes 
        a protein.</explanation>
      </question>
      <question number="1" title="Biology #1" type="0">
        <images>
        </images>
        <category>Biology</category>
        <description>&lt;b&gt;The site of cellular respiration is:&lt;/b&gt;
        </description>
        <choices>
          <choice name="A">DNA polymerase</choice>
          <choice name="B">Ribosomes</choice>
          <choice correct_answer="true" name="C">Mitochondria</choice>
          <choice name="D">RNA polymerase</choice>
          <choice name="E">Vacuoles</choice>
        </choices>
        <explanation>&lt;b&gt;Answer:&lt;/b&gt; C, Mitochondria.  The 
        mitochondrion (plural mitochondria) is known as the &#8220;
        powerhouse&#8221; of the cell for its role in energy production.
        &lt;br /&gt;&lt;br /&gt;&lt;b&gt;Key Takeaway: &lt;/b&gt;The 
        mitochondrion is a membrane-bound organelle found in most 
        eukaryotic cells.  The dominant role of the mitochondrion is the 
        production of ATP through cellular respiration, which is dependent 
        on the presence of oxygen.  All forms of cellular respiration, 
        glycolysis, Krebs&#8217; cycle, and oxidative phosphorylation, 
        take place within the mitochondria.</explanation>
      </question>
    </questions>
  </app>
</root>

While the output that I want is this

<root>
  <categories>
     <category name="Biology" />
   </categories>
  <app>
    <mainHeader><![CDATA[AP Biology]]></mainHeader>
    <questions>
       <question type="0" number="1" title="Biology #1">
        <images />
        <category><![CDATA[Biology]]></category>
        <description><![CDATA[<b>Which of the following is the 
        site of protein synthesis?</b>]]></description>
        <choices>
          <choice name="A"><![CDATA[Cell wall]]></choice>
          <choice name="B" correct_answer="true"><![CDATA[Ribosomes]]></choice>
          <choice name="C"><![CDATA[Vacuoles]]></choice>
          <choice name="D"><![CDATA[DNA polymerase]]></choice>
          <choice name="E"><![CDATA[RNA polymerase]]></choice>
        </choices>
        <explanation><![CDATA[<b>Answer:</b> B, Ribosomes.  Translation, 
        the process that converts mRNA code into protein, takes place in 
        ribosomes.<br /><br /><b>Key Takeaway: </b>Ribosomes are complexes 
        of RNA and protein that are located in cell nuclei.  Ribosomes 
        catalyze both the conversion of the mRNA code into amino acids as 
        well as the assembly of the individual amino acids into a peptide 
        change that becomes a protein.]]></explanation>
      </question>
      <question type="0" number="2" title="Biology #1">
        <images />
        <category><![CDATA[Biology]]></category>
        <description><![CDATA[<b>The site of cellular respiration 
        is:</b>]]></description>
        <choices>
          <choice name="A"><![CDATA[DNA polymerase]]></choice>
          <choice name="B"><![CDATA[Ribosomes]]></choice>
          <choice name="C" correct_answer="true"><![CDATA[Mitochondria]]></choice>
          <choice name="D"><![CDATA[RNA polymerase]]></choice>
          <choice name="E"><![CDATA[Vacuoles]]></choice>
        </choices>
        <explanation><![CDATA[<b>Answer:</b> C, Mitochondria.  The 
        mitochondrion (plural mitochondria) is known as the “powerhouse” 
        of the cell for its role in energy production.<br /><br />
        <b>Key Takeaway: </b>The mitochondrion is a membrane-bound 
        organelle found in most eukaryotic cells.  The dominant role 
        of the mitochondrion is the production of ATP through cellular 
        respiration, which is dependent on the presence of oxygen.  
        All forms of cellular respiration, glycolysis, Krebs’ cycle, 
        and oxidative phosphorylation, take place within the 
        mitochondria.]]></explanation>
      </question>
    </questions>
  </app>
</root>

How do I maintain the CDATA tags in my merged output? How do I keep the <b>, <br>, "" in my merged output instead of getting weird stuff like <b>? Sorry for my really noob questions, but I really appreciate the help.

Answer 1

CDATA is specifically for data that the xml parser should ignore. I think the best you'll be able to do in these circumstances, then, is to capture the text like so:

>>> element = et.fromstring('''<explanation><![CDATA[<b>Answer:</b> B, Ribosomes.  Translation, 
        the process that converts mRNA code into protein, takes place in 
        ribosomes.<br /><br /><b>Key Takeaway: </b>Ribosomes are complexes 
        of RNA and protein that are located in cell nuclei.  Ribosomes 
        catalyze both the conversion of the mRNA code into amino acids as 
        well as the assembly of the individual amino acids into a peptide 
        change that becomes a protein.]]></explanation>''')
>>> element.text
'<b>Answer:</b> B, Ribosomes.  Translation, \n        the process that converts mRNA code into protein, takes place in \n        ribosomes.<br /><br /><b>Key Takeaway: </b>Ribosomes are complexes \n        of RNA and protein that are located in cell nuclei.  Ribosomes \n        catalyze both the conversion of the mRNA code into amino acids as \n        well as the assembly of the individual amino acids into a peptide \n        change that becomes a protein.'

Then you can unescape your text as @praveen suggested.

Answer 2

Use the HTMLParse python library but this doesn't create those CDATA stuff.

text = """
<root>
  <categories>
    <category name="Biology" />
  </categories>
  <app>
    <mainHeader>AP Biology</mainHeader>
    <questions>
      <question number="1" title="Biology #1" type="0">
        <images />
        <category>Biology</category>
        <description>&lt;b&gt;Which of the following is the site 
        of protein synthesis?&lt;/b&gt;</description>
        <choices>
          <choice name="A">Cell wall</choice>
          <choice correct_answer="true" name="B">Ribosomes</choice>
          <choice name="C">Vacuoles</choice>
          <choice name="D">DNA polymerase</choice>
          <choice name="E">RNA polymerase</choice>
        </choices>
        <explanation>&lt;b&gt;Answer:&lt;/b&gt; B, Ribosomes.  
        Translation, the process that converts mRNA code into protein, 
        takes place in ribosomes.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;
        Key Takeaway: &lt;/b&gt;Ribosomes are complexes of RNA and protein 
        that are located in cell nuclei.  Ribosomes catalyze both the 
        conversion of the mRNA code into amino acids as well as the assembly 
        of the individual amino acids into a peptide change that becomes 
        a protein.</explanation>
      </question>
      <question number="1" title="Biology #1" type="0">
        <images>
        </images>
        <category>Biology</category>
        <description>&lt;b&gt;The site of cellular respiration is:&lt;/b&gt;
        </description>
        <choices>
          <choice name="A">DNA polymerase</choice>
          <choice name="B">Ribosomes</choice>
          <choice correct_answer="true" name="C">Mitochondria</choice>
          <choice name="D">RNA polymerase</choice>
          <choice name="E">Vacuoles</choice>
        </choices>
        <explanation>&lt;b&gt;Answer:&lt;/b&gt; C, Mitochondria.  The 
        mitochondrion (plural mitochondria) is known as the &#8220;
        powerhouse&#8221; of the cell for its role in energy production.
        &lt;br /&gt;&lt;br /&gt;&lt;b&gt;Key Takeaway: &lt;/b&gt;The 
        mitochondrion is a membrane-bound organelle found in most 
        eukaryotic cells.  The dominant role of the mitochondrion is the 
        production of ATP through cellular respiration, which is dependent 
        on the presence of oxygen.  All forms of cellular respiration, 
        glycolysis, Krebs&#8217; cycle, and oxidative phosphorylation, 
        take place within the mitochondria.</explanation>
      </question>
    </questions>
  </app>
</root>
"""

import HTMLParser
html_parser = HTMLParser.HTMLParser()
unescaped = html_parser.unescape(text)

print unescaped

Output:

<root>
  <categories>
    <category name="Biology" />
  </categories>
  <app>
    <mainHeader>AP Biology</mainHeader>
    <questions>
      <question number="1" title="Biology #1" type="0">
        <images />
        <category>Biology</category>
        <description><b>Which of the following is the site 
        of protein synthesis?</b></description>
        <choices>
          <choice name="A">Cell wall</choice>
          <choice correct_answer="true" name="B">Ribosomes</choice>
          <choice name="C">Vacuoles</choice>
          <choice name="D">DNA polymerase</choice>
          <choice name="E">RNA polymerase</choice>
        </choices>
        <explanation><b>Answer:</b> B, Ribosomes.  
        Translation, the process that converts mRNA code into protein, 
        takes place in ribosomes.<br /><br /><b>
        Key Takeaway: </b>Ribosomes are complexes of RNA and protein 
        that are located in cell nuclei.  Ribosomes catalyze both the 
        conversion of the mRNA code into amino acids as well as the assembly 
        of the individual amino acids into a peptide change that becomes 
        a protein.</explanation>
      </question>
      <question number="1" title="Biology #1" type="0">
        <images>
        </images>
        <category>Biology</category>
        <description><b>The site of cellular respiration is:</b>
        </description>
        <choices>
          <choice name="A">DNA polymerase</choice>
          <choice name="B">Ribosomes</choice>
          <choice correct_answer="true" name="C">Mitochondria</choice>
          <choice name="D">RNA polymerase</choice>
          <choice name="E">Vacuoles</choice>
        </choices>
        <explanation><b>Answer:</b> C, Mitochondria.  The 
        mitochondrion (plural mitochondria) is known as the “
        powerhouse” of the cell for its role in energy production.
        <br /><br /><b>Key Takeaway: </b>The 
        mitochondrion is a membrane-bound organelle found in most 
        eukaryotic cells.  The dominant role of the mitochondrion is the 
        production of ATP through cellular respiration, which is dependent 
        on the presence of oxygen.  All forms of cellular respiration, 
        glycolysis, Krebs’ cycle, and oxidative phosphorylation, 
        take place within the mitochondria.</explanation>
      </question>
    </questions>
  </app>
</root>