CODEWITHSUNDEEP
javaiojava.util.scanner
Swaranga Sarma
Swaranga Sarma

Reputation: 13423

java.util.Scanner malfunctioning while reading large files

I wrote a program where I used a Scanner to read lines from log files and parse each line to find something important. It is important that I read every line of the log file. I wrote the following piece of code to scan each line

Scanner s = new Scanner(new File("Large.log"));
while(s.hasNextLine())
{
    String line = s.nextLine();
    //do the processing of the log line
}

The above code behaves in a weird manner. It stops reading lines after a random number of lines [around after 1 million lines]. I modified the above code to check the last line read and also checked the log file using Notepad++. There were a lot of lines remaining in the file after that particular line. I added another System.out.println(s.hasNextLine()) after the end of the while loop and it prints false.

However if I try to do the above using a BufferedReader the program works fine. Is there any limitation with the util IO classes in Java?

Upvotes: 3

Views: 2929

Answers (2)

Kowshik
Kowshik

Reputation: 1641

I just dumped a string containing 50 characters to a temporary file, repeating the string 5 million times. And Scanner works fine for me when I try to read the file line by line.

I see two possible problems in your case :

  1. May be you are trying to read a huge line that passes Scanner's internal buffer size for reading a line ?
  2. Though unlikely, I hope there are no concurrent modifications to the same file by a different process/thread.

Upvotes: 0

Michael Aaron Safyan
Michael Aaron Safyan

Reputation: 95539

This sounds like a large file support issue with your particular JVM implementation. It is a common problem for a lot of standard file I/O to not work with files > 4 GB on 32-bit OSs. There are typically alternative versions of the file APIs to explicitly support large files, but the person implementing the JVM would have to remember to use those. Out of curiosity what OS are you using and is it 64-bit?

Upvotes: 3

Related Questions