CODEWITHSUNDEEP
java
Talen Kylon
Talen Kylon

Reputation: 1968

StreamTokenizer behavior

I'm in the midst of developing my own parser for a lab and I've run into some strange behavior with StreamTokenizer. It seems that anything surrounded by single quotes is getting skipped over.

Code

StreamTokenizer st = new StreamTokenizer(new FileReader("input.txt"));
boolean eof = false;

do{
   int i = 0;
   int token = st.nextToken();
   switch (token){
       case StreamTokenizer.TT_EOF:
            System.out.println("EOF");
            eof = true;
            break;
       case StreamTokenizer.TT_EOL:
            System.out.println("EOL");
            break;
       case StreamTokenizer.TT_WORD:
            System.out.println("Word: " + st.sval);
            break;
       case StreamTokenizer.TT_NUMBER:
            System.out.println("Number: " + st.nval);
            break;
       default:
            System.out.println((char) token + " encountered.");
            break;

   }
} while (!eof);

Input: 

top 'AT THE TOP' {
   l 2{ window{Open Up} } 
}

Output:

Word: top
' encountered.
{ encountered.
Word: l
Number: 2.0
{ encountered.
Word: window
{ encountered.
Word: Open
Word: Up
} encountered.
} encountered.
} encountered.
EOF

I noticed that I can set the quote char by using the following method:

st.quoteChar('\'');

I thought that this would allow me to set the quote char that, if encountered, the next token will be everything up until the next quote char.

Unfortunately that did not work as I thought, and now I am stuck.

Upvotes: 0

Views: 848

Answers (1)

Mike Samuel
Mike Samuel

Reputation: 120516

The javadoc says

public int ttype

For a quoted string token, its value is the quote character.

and the sval docs say

When the current token is a quoted string token, this field contains the body of the string.

so you need to have a

case '\'':
  System.out.println("Quoted value is " + st.sval);
  break;

or something similar to handle single quoted strings or reset the syntax so it's not treating ' as a quoteChar.

Upvotes: 3

Related Questions