Pyparsing operating on context

Question

How can I build a pyparsing program that allows operations being executed on a context/state object?

An example of my program looks like this:

load 'data.txt'
remove line 1
remove line 4

The first line should load a file and line 2 and 3 are commands that operate on the content of the file. As a result, I expect the content of the file after all commands have been executed.

load_cmd = Literal('load') + filename
remove_cmd = Literal('remove line') + line_no
more_cmd = ...

def load_action(s, loc, toks):
    # load file, where should I store it?

load_cmd.setParseAction(load_action)

def remove_line_action(s, loc, toks):
    # remove line, how to obtain data to operate on? where to write result?

remove_line_cmd.setParseAction(remove_cmd)

# Is this the right way to define a whole program, i.e. not only one line?
program = load_cmd + remove_cmd | more_cmd |...

# How do I obtain the result?
program.scanString("""
    load 'data.txt'
    remove line 1
    remove line 4
""")

Answer 1

I have written a few pyparsing examples of this command-parsing style, you can find them online at: http://pyparsing.wikispaces.com/file/view/simpleBool.py/451074414/simpleBool.py http://pyparsing.wikispaces.com/file/view/eval_arith.py/68273277/eval_arith.py

I have also written a simple Adventure-style game processor, which accepts parsed command structures and executes them against a game "world", which functions as the command executor. I presented this at PyCon 2006, but the link from the conference page has gone stale - you can find it now at http://www.ptmcg.com/geo/python/confs/pyCon2006_pres2.html (the presentation is written using S5 - mouse over the lower right corner to see navigation buttons). The code is at http://www.ptmcg.com/geo/python/confs/adventureEngine.py.txt, and UML diagram for the code is at http://www.ptmcg.com/geo/python/confs/pyparsing_adventure.pdf.

The general pattern I have found to work best is similar to the old Model-View-Controller pattern.

The Model is your virtual machine, which maintains the context from command to command. In simple_bool the context is just the inferred local variable scope, since each parsed statement is just evaled. In eval_arith, this context is kept in the EvalConstant._vars dict, containing the names and values of pre-defined and parsed variables. In the Adventure engine, the context is kept in the Player object (containing attributes that point to the current Room and the collection of Items), which is passed to the parsed command object to execute the command.

The View is the parser itself. It extracts the pieces of each command and composes an instance of a command class. The interface to the command class's exec method depends on how you have set up the Model. But in general you can envision that the exec method you define will take the Model as one of, if not its only, parameter.

Then the Controller is a simple loop that implements the following pseudo-code:

while not finished
    read a command, assign to commandstring

    parse commandstring, use parsed results to create commandobj (null if bad parse)

    if commandobj is not null:

        commandobj.exec(context)

    finished = context.is_finished()

If you implement your parser using pyparsing, then you can define your Command classes as subclasses of this abstract class:

class Command(object):
    def __init__(self, s, l, t):
        self.parameters = t
    def exec(self, context):
        self._do_exec(context)

When you define each command, the corresponding subclass can be passed directly as the command expression's parse action. For instance, a simplified GO command for moving through a maze would look like:

goExpression = Literal("GO") + oneOf("NORTH SOUTH EAST WEST")("direction")
goExpression.setParseAction(GoCommand)

For the abstract Command class above, a GoCommand class might look like:

class GoCommand(Command):
    def _do_exec(self, context):
        if context.is_valid_move(self.parameters.direction):
            context.move(self.parameters.direction)
        else:
            context.report("Sorry, you can't go " + 
                            self.parameters.direction + 
                            " from here.")

By parsing a statement like "GO NORTH", you would get back not a ParseResults containing the tokens "GO" and "NORTH", but a GoCommand instance, whose parameters include a named token "direction", giving the direction parameter for the GO command.

So the design steps to do this are:

• design your virtual machine, and its command interface

• create a class to capture the state/context in the virtual machine

• design your commands, and their corresponding Command subclasses

• create the pyparsing parser expressions for each command

• attach the Command subclass as a parse action to each command's pyparsing expression

• create an overall parser by combining all the command expressions using '|'

• implement the command processor loop

Answer 2

I would do something like this:

cmdStrs = '''
load
remove line
add line
some other command
'''

def loadParse(val): print 'Load --> ' + val
def removeParse(val): print 'remove --> ' + val
def addLineParse(val): print 'addLine --> ' + val
def someOtherCommandParse(val): print 'someOther --> ' + val


commands  = [ l.strip() for l in cmdStrs.split('\n') if l.strip() !='' ]
functions = [loadParse,
             removeParse,
             addLineParse,
             someOtherCommandParse]

funcDict = dict( zip(commands, functions) )

program = '''

    # This is a comment

    load 'data.txt' # This is another comment
    remove line 1
    remove line 4
'''

for l in program.split('\n'):
    l = l.strip().split('#')[0].strip() # remove comments
    if l == '': continue

    commandFound = False
    for c in commands:
        if c in l: 
            funcDict[c](l.split(c)[-1])
            commandFound = True

    if not commandFound:
        print 'Error: Unknown command : ', l 

Of course, you can put the entire thing within a class and make it an object, but you see the general structure. If you have an object, then you can go ahead and create a version which can handle contextual/state information. Then, the functions above will simply be member functions.

Why do I get a sense that you are starting on Python after learning Haskell? Generally people go the other way. In Python you get state for free. You don't need Classes. You can use classes to handle more than one state within the same program :).