CODEWITHSUNDEEP
javaheap-memory
user2324644
user2324644

Reputation: 43

InputStream read part of file and output

I have a set of large files ( about 2-3GB each) and I need to use InputStream to read it and output part of it to a new file.

//this is just for example
int size=100; 
String src="pathtofile";
OutputStream os = null; 
Inputstream is = new FileInputStream(new File(src));
byte[] buffer = new byte[size];
byte[] bufferis = getBytesFromIS(is); 

buffer=Arrays.copyOfRange(bufferis,0,buffer.length);

String tempstr=new String(buffer);  
byte[] tempBytes=Arrays.copyOfRange(bufferis, buffer.length,is.available());
os = new FileOutputStream(new File(dest));
copy(new ByteArrayInputStream(tempBytes), os); //function writing to file

This works fine with small files but when using on large files, I keep getting OutOfMemoryError, even when I set -Xmx6114m, still getting OutOfMemoryError.

Upvotes: 0

Views: 1751

Answers (2)

SaneResponder
SaneResponder

Reputation: 11

You don't show getBytesFromIS(is) but assuming that it reads the entire file, that's not necessary.

The easiest way is to use a library that already provides this functionality and is fully tested. For example, Apache Commons IOUtils.

Otherwise, you can use InputStream.read(b, off, len), starting from an offset of 0 and with your desired length. Beware, however, that this function is not guaranteed to read the entire amount of data that you want.

Upvotes: 1

maaartinus
maaartinus

Reputation: 46402

The maximum array length is slightly less than Integer.MAX_VALUE, i.e., about 2e9. So you can't read the whole into a byte[] and have to use something else. A ByteByffer is probably the fastest solution (memory mapped file).

Upvotes: 1

Related Questions