CODEWITHSUNDEEP
javafiletextlines
Grrace
Grrace

Reputation: 95

Reading group of lines in HUGE files

I have no idea how to do the following: I want to process a really huge textfile (almost 5 gigabytes). Since I cannot copy the file into temporarily memory, I thought of reading the first 500 lines (or as many as fit into the memory, I am not sure about that yet), do something with them, then go on to the next 500 until I am done with the whole file.

Could you post an example of the "loop" or command that you need for that? Because all the ways I tried resulted in starting from the beginning again but I want to go on after finishing the previous 500 lines.

Help appreciated.

Upvotes: 0

Views: 1011

Answers (2)

JJ.
JJ.

Reputation: 5475

Ok so you indicated in a comment above that you only want to keep certain lines, writing them to a new file based on certain logic. You can read in one line at a time, decide whether to keep it and if so write it to the new file. This approach will use very little memory since you are only holding one line at a time in memory. Here is one way to do that:

BufferedReader br = new BufferedReader(new FileReader(file));
String lineRead = null;
FileWriter fw = new FileWriter(new File("newfile.txt"), false);

while((lineRead = br.readLine()) != null)
{
     if (true) // put your test conditions here
     {
         fw.write(lineRead);
         fw.flush();
     }
}
fw.close();
br.close();

Upvotes: 0

Rocky Pulley
Rocky Pulley

Reputation: 23301

BufferedReader br = new BufferedReader(new FileReader(file));
String line = null;
ArrayList<String> allLines = new ArrayList<String>();

while((line = br.readLine()) != null) {
     allLines.add(line);
     if (allLines.size() > 500) {
          processLines(allLines);
          allLines.clear();
     }
}

processLines(allLines);

Upvotes: 1

Related Questions