CODEWITHSUNDEEP
flex-lexerlexical-analysis
Matteo
Matteo

Reputation: 14930

Strange flex behaviour

I have a scanner for a dice notation with the following scanner

%option debug
%option noyywrap
%option yylineno

%{    
#include <limits.h>
#include "parser.h"
%} 

%%

[0-9]+      {
    errno = 0;
#ifdef DEBUG
    printf("Scanner: NUMBER %s (%i)\n", yytext, yyleng);
#endif
    long number =  strtol( yytext, NULL, 10);
    if ( errno != 0 && errno != ERANGE && number == 0 ) {
        printf("Error: incorrect number %s\n", yytext);
        exit(EXIT_FAILURE);
    }
    // we only accept integers
    if ( number > INT_MAX ) {
        printf("Error: %s is too large\n", yytext );
        exit(EXIT_FAILURE);
    }
    yylval.int_type = (int)number;
    return NUMBER;
}

\+          { return PLUS;    }

\-          { return MINUS;   }

["*"x]      { return TIMES;   }

\/          { return DIV;     }

d|D         {
#ifdef DEBUG
    printf("Scanner: DICE\n");
#endif
    return DICE;
}

f|F         { return FUDGE;   }

h|H         { return HIGH;    }

l|L         { return LOW;     }

"("         { return LPAREN;  }

")"         { return RPAREN;  }

"{"         {
#ifdef DEBUG
    printf("Scanner: LCURLY\n");
#endif
    return LCURLY;
}

"}"         {
#ifdef DEBUG
    printf("Scanner: RCURLY\n");
#endif
    return RCURLY;
}

">"         { return GT; }
">="        { return GE; }
"<"         { return LT; }
"<="        { return LE; }
"!="        { return NE; }
"<>"        { return NE; }
"%"         { return PERCENT; }

,           {
#ifdef DEBUG
    printf("Scanner: COMMA\n");
#endif
    return COMMA;
}

[[:blank:]] {
    /* ignore spaces */
#ifdef DEBUG
    printf("Scanner: BLANK\n");
#endif
}

.           { printf("Error: unknown symbol '%s'\n", yytext); exit(EXIT_FAILURE); }

%%

When I parse something like 4{3d6, 1d5} everything works fine. But with 4{3d6,1d5} the scanner has a strange behaviour and misses the first curly bracket.

The debugging output is

--accepting rule at line 20 ("43")
Scanner: NUMBER 43 (2)
--accepting rule at line 47 ("d")
Scanner: DICE
--accepting rule at line 20 ("641")
Scanner: NUMBER 641 (3)
--accepting rule at line 47 ("d")
Scanner: DICE

The scanner is matching 4{3 as 43 although the { is not included in [0-9]+.

As the different behaviour is triggered by a blank much later in the expression I suspect that I have missed something in the space handling but I do not see why it should mess with the matching of an integer at the beginning of the expression.

Any hint?

Upvotes: 3

Views: 147

Answers (1)

Jay Kominek
Jay Kominek

Reputation: 8783

If a Unix shell is getting its hands on your input 4{3d6,1d5} would be expanded to 43d6 41d5. Your lexer completely ignores blanks, so that then becomes 43d641d5, which (modulo some truncation on your part?) is what you're reporting.

I duplicated your code, and when I run something like:

echo 4{3d6,1d5} | ./lex

I get your problem. If I run:

echo '4{3d6,1d5}' | ./lex

then everything is fine. It's also fine if I type 4{3d6,1d5} into a file, and then run the lexer on the file.

Upvotes: 3

Related Questions