how to preserve types when using pyparsing and returning the result as a dictionary

Question

I'm a newbie in pyparsing and hope somebody can help me.The type of text I am trying to parse is of the following structure:

I have key=value pairs in a line that can have one or more pairs. The values can be of many types such as string, int, float, list, dictionary. The key is always a string. Example of a line with 4 pairs:

mode='clip' clipzeros=True field='1331+0444=3C286' clipminmax=[0,1.2]

So I have defined my grammar and parser to be:

import pyparsing as pp

key = pp.Word(pp.alphas+"_")
separator = pp. Literal("=").suppress()
value = pp.Word(pp.printables)
pair = key+separator+value

line = pp.OneOrMore(pair)

mytest = "mode='clip' clipzeros=True field='1331+0444=3C286' clipminmax=[0,1.2]"
res = line.parseString(mytest)
print res

It returns this:

['mode', "'clip'", 'clipzeros', 'True', 'field', "'1331+0444=3C286'", 'clipminmax', '[0,1.2]']

Two things I want to have as results:

1. I'd like to have the results as a dictionary such as: {"mode":"clip", "clipzeros":True, "field":"1331+0444=3C286", "clipminmax":[0,1.2]}

2. I'd like to keep the types of the values in the resulting dictionary. For example: the type of value clipzeros is a boolean. The type of value clipminmax is a list.

Is this at all possible in pyparsing?

Thanks a lot for any help.

Sandra

Answer 1

Try using eval() to get your type.

import pyparsing as pp

key = pp.Word(pp.alphas+"_")
separator = pp. Literal("=").suppress()
value = pp.Word(pp.printables)
pair = key+separator+value

line = pp.OneOrMore(pair)

mytest = "mode='clip' clipzeros=True field='1331+0444=3C286' clipminmax=[0,1.2]"
res = line.parseString(mytest)
mydict = dict(zip(res[::2],[eval(x) for x in res[1::2]])

Yields:

{'field': '1331+0444=3C286', 'mode': 'clip', 'clipzeros': True, 'clipminmax': [0, 1.2]}

Another Example:

res = ['mode', "'clip'", 'clipzeros', 'True', 'field', "'R RQT'", 'clipminmax', '[0,1.2]']
mydict = dict(zip(res[::2],[eval(x) for x in res[1::2]]))

print mydict

Yields:

  {'field': 'R RQT', 'mode': 'clip', 'clipzeros': True, 'clipminmax': [0, 1.2]}

ALTERNATIVE to pyparser (I don't have that module):

class Parser():
    def __init__(self,primarydivider,secondarydivider):
        self.prime = primarydivider
        self.second = secondarydivider

    def parse(self,string):
        res = self.initialsplit(string)
        new = []
        for entry in res:
            if self.second not in entry:
                new[-1] += ' ' + entry
            else:
                new.append(entry)
        return dict((entry[0],eval(entry[1])) for entry in [entry.split(self.second) for entry in new])

    def initialsplit(self,string):
        return string.split(self.prime)

mytest = "mode='clip' clipzeros=True field='AEF D' clipminmax=[0,1.2]"
myParser = Parser(' ', '=')
parsed = myParser.parse(mytest)
print parsed

Yields

 {'field': 'AEF D', 'mode': 'clip', 'clipzeros': True, 'clipminmax': [0, 1.2]}

OP's Comments Edit:

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> mytest = "mode='clip' clipzeros=True field='R RQT' clipminmax=[0,1.2]"
>>> print mytest
mode='clip' clipzeros=True field='R RQT' clipminmax=[0,1.2]
>>>

Answer 2

Rather than use something as generic as Word(printables), I'd suggest defining specific expressions for different types of value literals:

from pyparsing import *

# what are the different kinds of values you can get?
int_literal = Combine(Optional('-') + Word(nums))
float_literal = Regex(r'\d+\.\d*')
string_literal = quotedString
bool_literal = Keyword("True") | Keyword("False")
none_literal = Keyword("None")
list_literal = originalTextFor(nestedExpr('[',']'))
dict_literal = originalTextFor(nestedExpr('{','}'))

# define an overall value expression, of the different types of values
value = (float_literal | int_literal | bool_literal | none_literal | 
        string_literal | list_literal | dict_literal)

key = Word(alphas + '_')

def evalValue(tokens):
    import ast
    # ast.literal_eval is safer than global eval()
    return [ast.literal_eval(tokens[0])]
pair = Group(key("key") + '=' + value.setParseAction(evalValue)("value"))

line = OneOrMore(pair)

Now parse your sample:

sample = """mode='clip' clipzeros=True field='1331+0444=3C286' clipminmax=[0,1.2]"""

result = line.parseString(sample)
for r in result:
    print r.dump()
    print r.key
    print r.value
    print

Prints:

['mode', '=', 'clip']
- key: mode
- value: clip
mode
clip

['clipzeros', '=', True]
- key: clipzeros
- value: True
clipzeros
True

['field', '=', '1331+0444=3C286']
- key: field
- value: 1331+0444=3C286
field
1331+0444=3C286

['clipminmax', '=', [0, 1.2]]
- key: clipminmax
- value: [0, 1.2]
clipminmax
[0, 1.2]