Writing a Python tool to convert XML to Python?

Question

I've been asked to look in to the possibility of converting XML to Python, so that the XML can be phased out of the current process & they'd be replaced with new python scripts.

Currently the XML files are used by python & ASP and look essentially like this;

<?xml version="1.0" encoding="UTF-8"?>
<script>
    <stage id="stage1" nextStage="stage2">
        <initialise>
            <variable id="year" value="2012" type="Integer"/>
            <executeMethod class="engineclass_date" method="getTodayAsString">
                <arguments>
                    <variable id="someVar"/>
                </arguments>
                <return>
                    <variable id="todaysDate"/>
                </return>
            </executeMethod>

Thanks to Pepr I've not got a parser/compiler in the works that produces Python code from the XML. It still needs a lot of work to handle all the possible elements, but it works, and hopefully others can learn from this like I am!

Still need to find a way to write a separate file for each stage of the script in order for the indent to function properly and handle errors properly.

Answer 1

Use the standard xml.etree.Element tree to extract the information from XML to Python objects (or the more enhanced third party lxml with the same API).

I recommend to read the Mark Pilrim's Dive Into Python 3, Chapter 12. XML (http://getpython3.com/diveintopython3/xml.html).

Here is the core of how the parser/compiler could be written. The idea is to go recursively through the elements, collect the neccessary information and output the code when it is possible:

import xml.etree.ElementTree as ET

class Parser:

    def __init__(self):
        self.output_list = []  # collected output lines
        self.il = 0            # indentation level


    def __iter__(self):
        return iter(self.output_list)


    def out(self, s):
        '''Output the indented string to the output list.'''
        self.output_list.append('    ' * self.il + s)


    def indent(self, num=1):
        '''Increase the indentation level.'''
        self.il += num


    def dedent(self, num=1):
        '''Decrease the indentation level.'''
        self.il -= num


    def parse(self, elem):
        '''Call the parser of the elem.tag name.

        The tag name appended to "parse_" and then the name of that
        function is called.  If the function is not defined, then
        self.parse_undefined() is called.'''

        fn_name = 'parse_' + elem.tag
        try:
            fn = getattr(self, fn_name)
        except AttributeError:
            fn = self.parse_undefined
        return fn(elem)


    def loop(self, elem):
        '''Helper method to loop through the child elements.'''
        for e in elem:
            self.parse(e)


    def parseXMLfile(self, fname):
        '''Reads the XML file and starts parsing from the root element.'''
        tree = ET.parse(fname)
        script = tree.getroot()
        assert script.tag == 'script'
        self.parse(script)


    ###################### ELEMENT PARSERS #######################

    def parse_undefined(self, elem):
        '''Called for the element that has no parser defined.'''
        self.out('PARSING UNDEFINED for ' + elem.tag)


    def parse_script(self, elem):
        self.loop(elem)


    def parse_stage(self, elem):
        self.out('')
        self.out('Parsing the stage: ' + elem.attrib['id'])
        self.indent()
        self.loop(elem)
        self.dedent()


    def parse_initialise(self, elem):
        self.out('')
        self.out('#---------- ' + elem.tag + ' ----------')
        self.loop(elem)


    def parse_variable(self, elem):
        tt = str   # default type
        if elem.attrib['type'] == 'Integer': 
            tt = int
        # elif ... etc for other types

        # Conversion of the value to the type because of the later repr().
        value = tt(elem.attrib['value'])  

        id_ = elem.attrib['id']

        # Produce the line of the output.
        self.out('{0} = {1}'.format(id_, repr(value)))


    def parse_execute(self, elem):
        self.out('')
        self.out('#---------- ' + elem.tag + ' ----------')
        self.loop(elem)


    def parse_if(self, elem):
        assert elem[0].tag == 'condition'
        condition = self.parse(elem[0])
        self.out('if ' + condition + ':')
        self.indent()
        self.loop(elem[1:])
        self.dedent()


    def parse_condition(self, elem):
        assert len(elem) == 0
        return elem.text


    def parse_then(self, elem):
        self.loop(elem)


    def parse_else(self, elem):
        self.dedent()
        self.out('else:')
        self.indent()
        self.loop(elem)


    def parse_error(self, elem):
        assert len(elem) == 0
        errorID = elem.attrib.get('errorID', None)
        fieldID = elem.attrib.get('fieldID', None)
        self.out('error({0}, {1})'.format(errorID, fieldID))


    def parse_setNextStage(self, elem):
        assert len(elem) == 0
        self.out('setNextStage --> ' + elem.text)


if __name__ == '__main__':
    parser = Parser()
    parser.parseXMLfile('data.xml')
    for s in parser:
        print s

When used with the data pasted here http://pastebin.com/vRRxfWiA, the script produces the following output:

Parsing the stage: stage1

    #---------- initialise ----------
    taxyear = 2012
    taxyearstart = '06/04/2012'
    taxyearend = '05/04/2013'
    previousemergencytaxcode = '747L'
    emergencytaxcode = '810L'
    nextemergencytaxcode = '810L'

    ...

    maxLimitAmount = 0
    PARSING UNDEFINED for executeMethod
    if $maxLimitReached$ == True:
        employeepayrecord = 'N'
        employeepayrecordstate = '2'
    else:
        employeepayrecordstate = '1'
    gender = ''
    genderstate = '1'
    title = ''
    forename = ''
    forename2 = ''
    surname = ''
    dob = ''
    dobinvalid = ''

    #---------- execute ----------
    if $dobstring$ != "":
        validDOBCheck = 'False'
        PARSING UNDEFINED for executeMethod
        if $validDOBCheck$ == False:
            error(224, dob)
        else:
            minimumDOBDate = ''
            PARSING UNDEFINED for executeMethod
            validDOBCheck = 'False'
            PARSING UNDEFINED for executeMethod
            if $validDOBCheck$ == False:
                error(3007161, dob)
        if $dobstring$ == "01/01/1901":
            error(231, dob)
    else:
        error(231, dob)

Parsing the stage: stage2

    #---------- initialise ----------
    address1 = ''

    ...

Answer 2

If you have an appropriate XML Schema for these XML files, there are tools like GenerateDS which will generate python classes based on them.

That would allow you to load all of the files in memory, and have them as objects. How you then store that data elsewhere...well, you don't say what you want to do, but you could do anything you usually can with python.