CODEWITHSUNDEEP
javawhitespacetokenizejava-ioremoving-whitespace
H J
H J

Reputation: 369

Streamtokenizer, whitespaces

I would like to use StreamTokenizer to extract a name from a java file. I have set the whitespaces as commas 

        inputTokenizer.whitespaceChars(',', ',');

However when I parse the inputfile for a name( firstname lastname with a space in between) the tokenizer treats firstname as one token and lastname as another token. I would like both of them to be treated as the same token, how can I do this?

For Example "Billy Jean" is treated as two separate tokens(Billy - token1 Jean - token2) and I want it to be treated as one.

Thanks

Upvotes: 0

Views: 2787

Answers (1)

arcy
arcy

Reputation: 13133

Your problem is that (evidently) spaces are regarded as delimiters by default (hardly surprising); you have set commas as whitespace characters, and so both commas and spaces are regarded as whitespace characters. The following program does what I think you want; note the line that sets spaces to be "wordChars".

import java.io.IOException;
import java.io.StreamTokenizer;
import java.io.StringReader;


public class TokenTeaser
{
  public static void main(String[] args)
  {
    try
    {
      String testString = "one two, three, four five";
      StringReader sr = new StringReader(testString);
      StreamTokenizer st = new StreamTokenizer(sr);
      st.whitespaceChars(',', ',');
      st.wordChars(' ', ' ');
      int currentToken = st.nextToken();
      while (currentToken != StreamTokenizer.TT_EOF)
      {
        System.out.println(st.sval);
        currentToken = st.nextToken();
      }
    }
    catch (IOException e)
    {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }

  }

}

Upvotes: 3

Related Questions