How to make flex use a text-containing file (.txt/.c/etc.) as its input

Question

I am completely new to flex, and my experience in programming is rather little. I need to create a scanner using flex that will output a stream of tokens eventually. For the moment, I just need to get the absolute basics up and running. I want the compiled output file "a.exe" to be able to be run from the text within a SINGLE file and not user input. The output should also be to a file. The assignment asks that the program is able to run like so in a cmd/PS window: .\a.exe inputfile.txt outputfile.txt Where input and output files are whatever file names are added in that order. As it stands currently, my program creates the output file I designate, but nothing is written to it. When trying to read the Flex Manual, I am very confused as I am still very new to computer sciences in general. As per the moment, I just want to get an executable file that will adhere to the rules section and output properly. This said I am generically just counting the characters in the input file and trying to display them to an output file. I also am trying to help the others in my class have a place to begin (as none of us were formally taught in this affair) so I am taking the time to try and create this file generically (with installation and usage instructions) so that I can give them a place to start the actual assignment of making the scanner.

I installed Flex 2.5.4a from http://gnuwin32.sourceforge.net/packages.html. I edited my Path to include the bin file after installation.

I build the file using the command "flex tokenout.l" and then "gcc lex.yy.c" and it generates an a.exe file. The file does not seem to work much at all past creating the output file.

code:

int num_lines = 0;
int num_chars = 0;
FILE *yyin;
FILE *yyout;

%%
\n  ++num_lines; ++num_chars;
.   ++num_chars;
%%

int yywrap(void) {
    return 0;
}
int main(int argc, char *argv[])
{
    yyin = fopen(argv[1],"r");
    yyout = fopen(argv[2],"w");
    yyparse();
    yylex();
    fprintf(yyout,"# of lines = %d, # of chars = %d\n", num_lines, num_chars);
    fclose(yyin);
    fclose(yyout);
    return 0;
}

The result should be that the line "# of lines = the actual # of lines, # of chars = the actual # of characters" to the file designated as the second argument.

Currently the file designated by the second argument is created but remains blank.

Answer 1

yyin = fopen(argv[1],"r");
yyout = fopen(argv[2],"w");
yyparse();
yylex();

As it stands currently, my program creates the output file I designate, but nothing is written to it.

You're confused because you don't know what your program is doing, and you don't know what it's doing because it's not telling you. What you need is feedback. In particular, you need to check for errors.

For example, what if the first fopen(3) fails? What if yyparse fails, or doesn't return? (It won't.) Check for errors, and have the program tell you what's happening.

#include <err.h>
if( argc < 3 ) {
    errx(EXIT_FAILURE, "syntax: foo in out");
}
if( (yyin = fopen(argv[1],"r")) == NULL ) {
    err(EXIT_FAILURE, "could not read '%s'", argv[1]);
}
if (yyout = fopen(argv[2],"w")) == NULL ) {
    err(EXIT_FAILURE, "could not write '%s'", argv[2]);
}

printf("starting yyparse\n");
if( 0 != yyparse() ) {
    errx(EXIT_FAILURE, "parse error");
}
printf("starting yylex\n");
if( 0 != yylex() )  {
    errx(EXIT_FAILURE, "lex error");
}

The above ensures the program is started with sufficient arguments, ensures both files are open successfully, and checks for errors parsing and lexing. That's just an example, though. As John Bollinger advised, you don't need yyparse because you're not using bison, and yyout controls only the file used by the flex ECHO statement. You can use your own global FILE * handle, and fprintf(3) to it in your flex actions.

What i think you will find is that you never see "starting yylex" on the screen, because yyparse never returns, because -- if it is being generated somewhere -- it's not returning, because it's calling yylex, which never returns anything to it.

I would delete those lines, and set flex debugging on with

  yy_flex_debug = 1;

before calling yylex. I think you'll find it makes more sense then.

Answer 2

Lex (flex) calls (or more precisely, generates code that calls) yywrap upon reaching the end of its input stream (in yyin). The job of this function is to:

1. Take care of closing the input file if needed / appropriate.
2. Switch to the next input file, if there is a next file.
3. Return nonzero (1, preferably) if flex should finish up, 0 if yyin is now re-opened to the next file.

Or, as the manual puts it:

When the scanner receives an end-of-file indication from YY_INPUT, it then checks the ‘yywrap()’ function. If ‘yywrap()’ returns false (zero), then it is assumed that the function has gone ahead and set up yyin to point to another input file, and scanning continues. If it returns true (non-zero), then the scanner terminates, returning 0 to its caller. Note that in either case, the start condition remains unchanged; it does not revert to INITIAL.

If you do not supply your own version of ‘yywrap()’, then you must either use ‘%option noyywrap’ (in which case the scanner behaves as though ‘yywrap()’ returned 1), or you must link with ‘-lfl’ to obtain the default version of the routine, which always returns 1.

(Modern flex has <<EOF>> rules which are generally a better way to deal with stacked input files, since transitions between files should almost always force a token boundary.)

Answer 3

You appear to be starting by adapting an example program from the Flex manual. That's fine, but maybe your very first step should be getting the exact example program working. After that, take it one step at a time. For example, the next step might be to get it to use the first argument as the name of the input file (and no other changes).

With respect to the partial program you have presented, I see two semantic issues:

1. When you use flex with bison (or yacc), it is the generated parser (accessed via yyparse()) that calls yylex(), and generally it will do so repeatedly until the input is exhausted. It is not useful in that case for the main program to call the lexer directly.

2. yyout is the file to which flex will direct the output of ECHO statements, nothing more, nothing less. It is not particularly useful to you, and I would ignore it for now.