CODEWITHSUNDEEP
javastream
yoto
yoto

Reputation: 21

How can I ensure that there's whitespace between specific tokens with StreamTokenizer?

I am working with StreamTokenizer to parse text, and I need to make sure that there's whitespace between certain tokens. For example, "5+5" is illegal, but "5 + 5" is valid.

I really don't know StreamTokenizer that well; I read the API but couldn't find anything to help me. How can I do this?

Upvotes: 0

Views: 180

Answers (2)

WhiteFang34
WhiteFang34

Reputation: 72049

Consider using Scanner instead since. E.g.:

String input = "5 + 5";
Scanner scanner = new Scanner(input).useDelimiter(" ");
int num1 = scanner.nextInt();
String op = scanner.next();
int num2 = scanner.nextInt();

This is a simplified example that assumes an input like "5+5" or "5 + 5". You should be checking scanner.hasNext() as you go, perhaps in a where loop to process more complicated inputs.

Upvotes: 0

ataylor
ataylor

Reputation: 66089

You'll have to set spaces as "ordinary characters". This means that they'll be returned as tokens on their own, but not folded into other tokens. Example:

StreamTokenizer st = new StreamTokenizer(new StringReader("5 + 5"));
st.ordinaryChar(32);
int tt = st.nextToken();  // tt = TT_NUMBER, st.nval = 5
tt = st.nextToken();      // tt = 32 (' ')
tt = st.nextToken();      // tt = 43 ('+')
tt = st.nextToken();      // tt = 32 (' ')
tt = st.nextToken();      // tt = TT_NUMBER, st.nval = 5
tt = st.nextToken();      // tt = TT_EOF

Unfortunately, you'll have to deal with whitespace tokens in your parser. I'd recommend rolling your own tokenizer. Unless you're doing something quick and dirty, StreamTokenizer is almost always the wrong choice.

Upvotes: 1

Related Questions