CODEWITHSUNDEEP
javaregexnewlinejava.util.scanner
SEK
SEK

Reputation: 1222

Java Scanner newline parsing with regex (Bug?)

I'm developing a syntax analyzer by hand in Java, and I'd like to use regex's to parse the various token types. The problem is that I'd also like to be able to accurately report the current line number, if the input doesn't conform to the syntax.

Long story short, I've run into a problem when I try to actually match a newline with the Scanner class. To be specific, when I try to match a newline with a pattern using the Scanner class, it fails. Almost always. But when I perform the same matching using a Matcher and the same source string, it retrieves the newline exactly as you'd expect it too. Is there a reason for this, that I can't seem to discover, or is this a bug, as I suspect?

FYI: I was unable to find a bug in the Sun database that describes this issue, so if it is a bug, it hasn't been reported.

Example Code:

Pattern newLinePattern = Pattern.compile("(\\r\\n?|\\n)", Pattern.MULTILINE);
String sourceString = "\r\n\n\r\r\n\n";
Scanner scan = new Scanner(sourceString);
scan.useDelimiter("");
int count = 0;
while (scan.hasNext(newLinePattern)) {
    scan.next(newLinePattern);
    count++;
}
System.out.println("found "+count+" newlines"); // finds 7 newlines
Matcher match = newLinePattern.matcher(sourceString);
count = 0;
while (match.find()) {
    count++;
}
System.out.println("found "+count+" newlines"); // finds 5 newlines

Upvotes: 4

Views: 5218

Answers (4)

polygenelubricants
polygenelubricants

Reputation: 383676

Your useDelimiter() and next() combo is faulty. useDelimiter("") will return 1-length substring on next(), because an empty string does in fact sit between every two characters.

That is, because "\r\n".equals("\r" + "" + "\n") so "\r\n" are in fact two tokens, "\r" and "\n", delimited by "".

To get the Matcher-behavior, you need findWithinHorizon, which ignores delimiters.

    Pattern newLinePattern = Pattern.compile("(\\r\\n?|\\n)", Pattern.MULTILINE);
    String sourceString = "\r\n\n\r\r\n\n";
    Scanner scan = new Scanner(sourceString);
    int count = 0;
    while (scan.findWithinHorizon(newLinePattern, 0) != null) {
        count++;
    }
    System.out.println("found "+count+" newlines"); // finds 5 newlines

API links

  • findWithinHorizon(Pattern pattern, int horizon)

    Attempts to find the next occurrence of the specified pattern [...] ignoring delimiters [...] If no such pattern is detected then the null is returned [...] If horizon is 0, then [...] this method continues to search through the input looking for the specified pattern without bound.

Related questions

Upvotes: 6

John Kugelman
John Kugelman

Reputation: 361556

When you use the Scanner with a delimiter of "" it will produce tokens that are each one character long. This is before your new line regex is applied. It then matches each of these characters against the new line regex; each one matches, so it produces 7 tokens. However, because it split the string into 1-character tokens it will not group adjacent \r\n characters into one token.

Upvotes: 0

Chris
Chris

Reputation: 10435

That is, in fact, the expected behaviour of both. The scanner primarily cares about splitting things into tokens using your delimiter. So it (lazily) takes your sourceString and sees it as the following set of tokens: \r, \n, \n, \r, \r, \n, and \n. When you then call hasNext it checks if the next token matches your pattern (which they all trivially do thanks to the ? on the \r\n?). The while loop therefore iterates over each of the 7 tokens.

On the other hand, the matcher will match the regex greedily - so it bundles the \r\ns together as you expect.

One way to emphasise the behaviour of Scanner is to change your regexp to (\\r\\n|\\n). This results in a count of 0. This is because the scanner reads the first token as \r (not \r\n), and then notices it doesn't match your pattern, so returns false when you call hasNext.

(Short version: the scanner tokenises using your delimiter before using your token pattern, the matcher doesn't do any form of tokenising)

Upvotes: 3

jasonmp85
jasonmp85

Reputation: 6817

It might be worth mentioning that your example is ambiguous. It could be:

\r
\n
\n
\r
\r
\n
\n

(seven lines)

or:

\r\n
\n
\r
\r\n
\n

(five lines)

The ? quantifier you have used is a greedy quantifier, which would probably make five the right answer, but because Scanner iterates over tokens (in your case individual characters, due to the delimiting pattern you chose), it will match reluctantly, one character at a time, arriving at the incorrect answer of seven.

Upvotes: 2

Related Questions