CODEWITHSUNDEEP
javamemorystreaminteger
Danny Brown
Danny Brown

Reputation: 9

Read a huge file of numbers in Java in a memory-efficient way?

In Java, having a file of 335Gb size that contains individual numbers at each line, I need to read it line by line like if it was a stream of numbers - I must not keep all the data in memory. I was told that Scanner class will not work. Could you please recommend the best possible way to do that?

Upvotes: 0

Views: 415

Answers (2)

Peter Lawrey
Peter Lawrey

Reputation: 533880

If you use BufferedReader you should be able to get up to 90 MB/s in one thread.

You can use trick to break up the file and read portion of the data concurrently, but this will only help if your disk read through put is high.

For example you can memory map 335 GB into memory at once without using the heap much. This will work even if you have a fraction of this amount of main memory.

What is the read transfer rate you can get with your disk subsystem?

Upvotes: 1

Joe
Joe

Reputation: 31

None of the java.io input stream classes would "keep all the data in memory". I think you are free to choose what is best for you such as BufferedReader or DataInputStream etc.

Upvotes: 3

Related Questions