PLY Lex and Yacc Issues

Question

I'm having trouble using PLY. I've been reading the documentation and decided to try out the examples. The lexing example worked perfectly but parsing I could not get to work. I also, looking at the docs, don't understand how you interface lex and yacc together to create a proper compiler. The parser only included the lexer's possible tokens and as far as I can tell nothing else.

I added a few things like colour (Colorama module) and slightly different messages but apart from that this is identical to the example:

#!/usr/bin/env python

### LEXICAL ANALYSIS ###


import ply.lex as lex

import colorama
colorama.init()

tokens = (
    "NUMBER",
    "PLUS",
    "MINUS",
    "MULTIPLY",
    "DIVIDE",
    "LBRACKET",
    "RBRACKET"
)


t_PLUS = r"\+"
t_MINUS = r"-"
t_MULTIPLY = r"\*"
t_DIVIDE = r"/"
t_LBRACKET = r"\("
t_RBRACKET = r"\)"

t_ignore = "\t\r "

def t_NUMBER(t):
    r"\d+"
    t.value = int(t.value)
    return t

def t_newline(t):
    r"\n+"
    t.lexer.lineno += len(t.value)

def t_COMMENT(t):
    r"\#.*"
    print "Comment:", t.value

def t_error(t):
    print colorama.Fore.RED + "\n\nLEXICAL ERROR: line", t.lexer.lineno, "and position", t.lexer.lexpos, "invalid token:", t.value.split("\n")[0] + colorama.Fore.RESET
    t.lexer.skip(len(t.value))


def mylex(inp):
    lexer = lex.lex()

    lexer.input(inp)


    for token in lexer:
        print "Token:", token

That works fine, but the parser, however, does not:

#!/usr/bin/env python


import ply.yacc as yacc

from langlex import tokens

def p_expression_plus(p):
    "expression : expression PLUS term"
    p[0] = p[1] + p[3]

def p_expression_minus(p):
    "expression : expression MINUS term"
    p[0] = p[1] - p[3]

def p_expression_term(p):
    "expression : term"
    p[0] = p[1]

def p_term_times(p):
    "term : term MULTIPLY factor"
    p[0] = p[1] * p[3]

def p_term_div(p):
    "term : term DIVIDE factor"
    p[0] = p[1] / p[3]

def p_term_factor(p):
    "term : factor"
    p[0] = p[1]

def p_factor_num(p):
    "factor : NUMBER"
    p[0] = p[1]

def p_factor_expr(p):
    "factor : LBRACKET expression RBRACKET"
    p[0] = p[2]

def p_error(p):
    print "Syntax error!"

parser = yacc.yacc()

while True:
    s = raw_input("calc > ")
    if not(s):
        continue
    result = parser.parse(s)
    print result

I get the following error when I try to run it:

calc > 5 + 10
Traceback (most recent call last):
  File "C:\Users\Max\Desktop\lang\langyacc.py", line 49, in <module>
    result = parser.parse(s)
  File "C:\Python27\lib\site-packages\ply\yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "C:\Python27\lib\site-packages\ply\yacc.py", line 881, in parseopt_notrack
    lexer = lex.lexer
AttributeError: 'module' object has no attribute 'lexer'

I'm a beginner to lex, yacc and compiler development in general and have no clue why this is happening. Any help would be appreciated.

Answer 1

You haven't built the lexer in your lexer file. You have it in the function mylex() but it doesn't actually get built.

Pull it out of the function.

lexer = lex.lex()

def mylex(inp):
    lexer.input(inp)
    # etc.