CODEWITHSUNDEEP
cbisonyaccflex-lexer
Derek
Derek

Reputation: 11915

Flex giving "fatal scanner internal error--end of buffer missed"

I am trying to use flex against a buffer that keeps getting filled and refilled.

I have multiple threads working with their own independent copies of the buffer.

In psuedo it looks something like this in my main program:

while(forever)
{
-fill buffer buf-
buf[length+1] = '\0';
buf[length+2] = '\0';
yy_scan_bytes(&buf, length+2);
yylex();
}

where multiple threads are running that loop, and all own their copy of buf.

I have a rule in my lex file that looks like this:

<<EOF>> {yyterminate(); yy_delete_buffer(YY_CURRENT_BUFFER);}

in an attempt to clean up after the buffer has run.

I am not sure if I am doing this right as this is my first attempt at using a flex scanner in conjunction wtih another program. Would I be better off doing all the buffer handling from within flex, and linking it with an extern declaration in my main program?

Thanks

Upvotes: 3

Views: 6012

Answers (1)

Chris Dodd
Chris Dodd

Reputation: 126175

The problem is that flex's default scanner is not reentrant -- it stores a bunch of info (including the current buffer to read from) in global variables, so if you try to have multiple threads scanning stuff at the same time, they'll step all over each other.

You need to use %option reentrant to create a reentrant scanner, and then create a separate yyscan_t for each thread to use, which you initialize with yylex_init and then pass as an extra argument to both yylex and yy_scan_bytes.

Also, yyterminate does an immediate return from the scanner, so your yy_delete_buffer call is never made. You need to call it after the scanner returns:

yyscan_t local;
yylex_init(&local);
while(forever)
{
    -fill buffer buf-
    buf[length+1] = '\0';
    buf[length+2] = '\0';
    YY_BUUFER_STATE bs = yy_scan_bytes(&buf, length+2, local);
    yylex(local);
    yy_delete_buffer(bs, local);
}

Upvotes: 3

Related Questions