CODEWITHSUNDEEP
javastreamtext-files
Steakie
Steakie

Reputation: 57

Reading huge textfiles Java

I've got a .txt with 1 Billion digits of pi. I read in the file as a String but i get an OutOfMemoryError. It worked with 1 MIllion digits tho. I save the String as a char[] array. Is it possible to somehow stream the .txt when i cycle through the whole array? I simply need a way to have access to all the 1 Billion digits as an array.

Upvotes: 3

Views: 115

Answers (4)

nafas
nafas

Reputation: 5423

According to doc

You should be able to get a String of length Integer.MAX_VALUE (always 2147483647 (231 - 1) by the Java specification, the maximum size of an array, which the String class uses for internal storage) or half your maximum heap size (since each character is two bytes), whichever is smaller

This is why you get the Exception,

if you don't really need the whole 1B chars. you can try using buffer which doesn't load the whole thing into memory.

BufferedReader br = new BufferedReader(new FileReader(new File("path to file")));
char[] data=new char[1000000] ;//however many chars you want;
int i=0;
while ((c = br.read()) != -1 && i<data.length) {
    data[i++]= c;
}

br.close();

Upvotes: 0

Jeremy Grand
Jeremy Grand

Reputation: 2370

You can open the file with a FileInputStream, and read it byte[] per byte[] to avoid the OOMError.

Upvotes: 0

Ghislain Fourny
Ghislain Fourny

Reputation: 7279

It is not only possible: it is both highly recommended and done in practice. What is usually done is to reuse the same kind of interfaces as Java libraries (InputStream, etc).

In this case, this could mean a new IntegerInputStream class that outputs the digits as a stream. This class can itself forward calls to FileInputStream. Internally, you can use char[] arrays to store the buffer and improve performance, or have calls directed via BufferedInputStream as Pavel suggests, but it is best to isolate the consumer from the internal buffer management and keep the appropriate level of abstraction to the use case (decimals of pi).

Upvotes: 0

Pavel Niedoba
Pavel Niedoba

Reputation: 1557

There is BufferedInputStream since java 1 or FileReader with 

public int read(char cbuf[], int offset, int length) throws IOException

I suggest you start from there

Upvotes: 4

Related Questions