Zip a great number of pdf file and download it: OutOfMemoryError: Java heap space

Question

I want create a zip file to download it. I have a great number of pdf file..

I need to zip them into sub-zip of 50MB one, so I create a master zip that contain all other sub-zip..

But when run my code I obtain:

Grave: Servlet.service() for servlet [appServlet] in context with path [/myProg] threw exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
java.lang.OutOfMemoryError: Java heap space

This is my code to divide and zip all files:

public void zippatore(List<File> filesDaZippare, ZipOutputStream masterZos, String zipName, String pdfName) throws IOException{
    float newSize = 0;
    boolean first = true;
    int count = 1;
    File file;
    ByteArrayOutputStream subBaos = new ByteArrayOutputStream();
    ZipOutputStream subZos = new ZipOutputStream(subBaos);

    masterZos.putNextEntry(new ZipEntry(zipName+" "+count++ +".zip"));
    while(filesDaZippare.size() > 0){
        file = filesDaZippare.remove(0);
        FileInputStream fis = new FileInputStream(file);
        newSize += file.length();
        String fileName = file.getName().substring(2);
        if(newSize < MAX_SIZE || first){
            subZos.putNextEntry(new ZipEntry(pdfName+" "+fileName+".pdf"));
            IOUtils.copy(fis, subZos);
            file.delete();
            subZos.closeEntry();
            first = false;
        }else{
            subZos.flush();
            subZos.close();

            masterZos.write(subBaos.toByteArray());
            masterZos.closeEntry();
            masterZos.putNextEntry(new ZipEntry(zipName+" "+count++ +".zip"));

            subZos = new ZipOutputStream(subBaos);
            subZos.putNextEntry(new ZipEntry(pdfName+" "+fileName+".pdf"));
            IOUtils.copy(fis, subZos);
            file.delete();

            newSize = file.length();
            first = true;
        }
        subZos.closeEntry();
    }
    subZos.closeEntry();
    subZos.flush();
    subZos.close();
    masterZos.write(subBaos.toByteArray());

    masterZos.closeEntry();
}

How can I fix it??

EDIT: I added the code to save in a tmp folder my sub-zip file.. And than I do this:

    File folder = new File(VariabiliGlobali.PATH_TMP);
    byte[] buffer = new byte[1024];

    for(File file : folder.listFiles()) {
        FileInputStream fis = new FileInputStream(file);
        masterZos.putNextEntry(new ZipEntry(file.getName()));

        int length;
        while ((length = fis.read(buffer)) > 0) {
            masterZos.write(buffer, 0, length);
        }

        masterZos.closeEntry();
        fis.close();
    }
    masterZos.flush();
    masterZos.close();

to get all zip file and put in my master zip file, but I obtain the same error when I put them in it...

Answer 1

From the code it can be seen that you are Zipping each of the sub zip into an outputstream backed by a ByteArrayOutput stream, which will consume memory. Writing each of the Sub file into a temporary location on the disk and then pushing the individual zip into the master zip would reduce your memory need, at the same time writing into the disk has an overhead, but if the overall size is not known this would be a better approach.

Writing your temporary zip/subzip to a FileOutputstream instead of a ByteArrayOutputStream would allow you to continue as long as the disk has enough space to handle it.

public void zippatore(List<File> filesDaZippare, ZipOutputStream masterZos, String   zipName, String pdfName) throws IOException{
    float newSize = 0;
    boolean first = true;
    int count = 1;
    File file;
    ByteArrayOutputStream subBaos = new ByteArrayOutputStream();
    ZipOutputStream subZos = new ZipOutputStream(subBaos);

    masterZos.putNextEntry(new ZipEntry(zipName+" "+count++ +".zip"));
    while(filesDaZippare.size() > 0){
    file = filesDaZippare.remove(0);
    FileInputStream fis = new FileInputStream(file);
    newSize += file.length();
    String fileName = file.getName().substring(2);
    if(newSize < MAX_SIZE || first){
        subZos.putNextEntry(new ZipEntry(pdfName+" "+fileName+".pdf"));
        IOUtils.copy(fis, subZos);
        file.delete();
        subZos.closeEntry();
        first = false;
    }else{
        subZos.flush();
        subZos.close();

        masterZos.write(subBaos.toByteArray());
        masterZos.closeEntry();
        masterZos.putNextEntry(new ZipEntry(zipName+" "+count++ +".zip"));
        subBaos = new ByteArrayOutputStream();
        subZos = new ZipOutputStream(subBaos);
        subZos.putNextEntry(new ZipEntry(pdfName+" "+fileName+".pdf"));
        IOUtils.copy(fis, subZos);
        file.delete();

        newSize = file.length();
        first = true;
    }
    subZos.closeEntry();
}
subZos.closeEntry();
subZos.flush();
subZos.close();
masterZos.write(subBaos.toByteArray());

masterZos.closeEntry();
}

Answer 2

I solved in this way: for master zip file I used a FileOutputStream, while to all others sub-zip files I use a ByteArrayOutputStream and I flushed and close it every 50 MB.

So I don't use a file to every sub-zip, and so using a stream is a bit quickly than use a file..

This is the code:

    FileOutputStream masterBos = new FileOutputStream(VariabiliGlobali.PATH_TMP+"Tmp.zip");
    ZipOutputStream masterZos = new ZipOutputStream(masterBos);


public void zippatore(List<File> filesDaZippare, ZipOutputStream masterZos, String zipName, String pdfName) throws IOException{
    float newSize = 0;
    boolean first = true;
    int count = 1;
    File file;
    ByteArrayOutputStream subBaos = new ByteArrayOutputStream();
    ZipOutputStream subZos = new ZipOutputStream(subBaos);

    masterZos.putNextEntry(new ZipEntry(zipName+" "+count++ +".zip"));
    while(filesDaZippare.size() > 0){
        file = filesDaZippare.remove(0);
        FileInputStream fis = new FileInputStream(file);
        newSize += file.length();
        String fileName = file.getName().substring(2);
        if(newSize < MAX_SIZE || first){
            subZos.putNextEntry(new ZipEntry(pdfName+" "+fileName+".pdf"));
            IOUtils.copy(fis, subZos);
            file.delete();
            subZos.closeEntry();
            first = false;
        }else{
            subZos.flush();
            subZos.close();
            subBaos.flush();
            subBaos.close();

            masterZos.write(subBaos.toByteArray());
            masterZos.closeEntry();
            masterZos.putNextEntry(new ZipEntry(zipName+" "+count++ +".zip"));

            subBaos = new ByteArrayOutputStream();
            subZos = new ZipOutputStream(subBaos);
            subZos.putNextEntry(new ZipEntry(pdfName+" "+fileName+".pdf"));
            IOUtils.copy(fis, subZos);
            file.delete();

            newSize = file.length();
            first = true;
        }
        subZos.closeEntry();
    }
    subZos.closeEntry();
    subZos.flush();
    subZos.close();
    masterZos.write(subBaos.toByteArray());

    masterZos.closeEntry();
}

Answer 3

Increase Java heap size by specifying VM arguments in your server configuration

Ex:

-Xmx1024m

I you are running Tomcat in eclipse you can specify VM arguments in Open launch configuration of Server Configuration.