CODEWITHSUNDEEP
javastringtokenizestringtokenizer
user697911
user697911

Reputation: 10531

How to get the delimiter from a StringTokenizer?

StringTokenizer tokenizer = new StringTokenizer(s, " ,.:;?![]'");

Is there a way to also retrieve the delimiter, in this case all thee punctuation marks?

For example, "This is a test, and is that a test too?"

I want the result of tokenization also includes the two tokens , and ?

Is that possible?

Upvotes: 2

Views: 2868

Answers (1)

Mureinik
Mureinik

Reputation: 311163

StringTokenizer has an overloaded constructor that takes a third boolean argument. Setting it to true will make the tokenzier return the delimiters too:

StringTokenizer tokenizer = new StringTokenizer(s, " ,.:;?![]'", true);
// Here ---------------------------------------------------------^

Upvotes: 3

Related Questions