how to create java zip archives with a max file size limit

Question

I need to write an algorithm in java (for an android app) to read a folder containing more folders and each of those containing images and audio files so the structure is this: mainDir/categorySubfolder/myFile1.jpg

My problem is that I need to limit the size of the archive to 16mb and at runtime, create as many archives as needed to contain all my files from my main mainDir folder.

I tried several examples from the net and I read the java documentation but I can't manage to understand and put it all together the way I need it. Has someone done this before or has a link or an example for me?

I resolved the reading of the files with a recursive method but I can't write the logic for the zip creation.

I'm open for suggestions or better a working example.

Answer 1

I am using below code/class to split and zip a large amount/size of files. I have tested this class on below

• number of uncompressed files : 116
• total size (uncompressed) : 29.1 GB
• ZIP file size limit (each) : 3 GB [MAX_ZIP_SIZE]
• total size (compressed) : 7.85 GB
• number of ZIP file (splited as MAX_ZIP_SIZE): 3

you have to change the value of MAX_ZIP_SIZE to 16(MB)10241024=16777216-22(zip header size)=16777194.
In my code, MAX_ZIP_SIZE set to 3 GB (ZIP has limitation of 4GB on various things).

final long MAX_ZIP_SIZE = 3221225472L; //3 GB

package com.company;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class QdeZip {

    public static String createZIP(String directoryPath, String zipFileName, String filesToZip) {
        try {
            final int BUFFER = 104857600; // 100MB
            final long MAX_ZIP_SIZE = 3221225472L; //3 GB
            long currentSize = 0;
            int zipSplitCount = 0;
            String files[] = filesToZip.split(",");
            if (!directoryPath.endsWith("/")) {
                directoryPath = directoryPath + "/";
            }
            byte fileRAW[] = new byte[BUFFER];
            ZipOutputStream zipOut = new ZipOutputStream(new FileOutputStream(directoryPath + zipFileName.toUpperCase()));
            ZipEntry zipEntry;
            FileInputStream entryFile;
            for (String aFile : files) {
                zipEntry = new ZipEntry(aFile);
                if (currentSize >= MAX_ZIP_SIZE) {
                    zipSplitCount++;
                    //zipOut.closeEntry();
                    zipOut.close();
                    zipOut = new ZipOutputStream(new FileOutputStream(directoryPath + zipFileName.toLowerCase().replace(".zip", "_" + zipSplitCount + ".zip").toUpperCase()));
                    currentSize = 0;
                }
                zipOut.putNextEntry(zipEntry);
                entryFile = new FileInputStream(directoryPath + aFile);

                int count;
                while ((count = entryFile.read(fileRAW, 0, BUFFER)) != -1) {
                    zipOut.write(fileRAW, 0, count);

                    //System.out.println("number of Bytes read = " + count);
                }
                entryFile.close();
                zipOut.closeEntry();
                currentSize += zipEntry.getCompressedSize();
            }

            zipOut.close();
            //System.out.println(directory + " -" + zipFileName + " -Number of Files = " + files.length);
        } catch (FileNotFoundException e) {
            return "FileNotFoundException = " + e.getMessage();
        } catch (IOException e) {
            return "IOException = " + e.getMessage();
        } catch (Exception e) {
            return "Exception = " + e.getMessage();
        }

        return "1";
    }
}

I have returned all Exception Messages as String to work with it. this my own case related to project.

Answer 2

As far as I can see How to split a huge zip file into multiple volumes? just suggests keeping track of the archive size so far, and when it approaches some arbitrary value (which should be lower than your max), it'll decide to start a new file. So for a 16MB limit, you could set the value to 10MB and start a new zip whenever this is reached, but if you reach 9MB and your next file zips down to 8MB, you'll end up with a zip bigger than your limit.

The code given in that post didn't seem to work for me because 1) it got the size before the ZipEntry was created, so it was always 0 and 2) it didn't write out any zip :-) If I've got that wrong - let me know.

The following works for me. For simplicity, I've taken it out of the Wrapper and just have it all in main(String args[]). There are many, many ways this code could be improved :-)

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.Deflater;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class ChunkedZipTwo {

    static final long MAX_LIMIT = 10 * 1000 * 1024; //10MB limit - hopefully this 

    public static void main(String[] args) throws Exception {
        String[] files = {"file1", "file2", "file3"};
        int i = 0;
        boolean needNewFile = false;
        long overallSize = 0;
        ZipOutputStream out = getOutputStream(i);
        byte[] buffer = new byte[1024];

        for (String thisFileName : files) {
            if (overallSize > MAX_LIMIT) {
                out.close();
                i++;
                out = getOutputStream(i);
                overallSize = 0;
            }

            FileInputStream in = new FileInputStream(thisFileName);
            ZipEntry ze = new ZipEntry(thisFileName);
            out.putNextEntry(ze);
            int len;
            while ((len = in.read(buffer)) > 0) {
                out.write(buffer, 0, len);
            }
            out.closeEntry();
            in.close();
            overallSize += ze.getCompressedSize();
        }
        out.close();
    }

    public static ZipOutputStream getOutputStream(int i) throws IOException {
        ZipOutputStream out = new ZipOutputStream(new FileOutputStream("bigfile" + i + ".zip"));
        out.setLevel(Deflater.DEFAULT_COMPRESSION);
        return out;
    }
}   

Answer 3

zip4j is a great library that can create multi-part zip files.

net.lingala.zip4j.core.ZipFile zipFile = new ZipFile("out.zip");
ZipParameters parameters = new ZipParameters();
parameters.setCompressionMethod(Zip4jConstants.COMP_DEFLATE);
parameters.setCompressionLevel(Zip4jConstants.DEFLATE_LEVEL_NORMAL);
zipFile.createZipFileFromFolder("path/to/source/dir", parameters, true, maximum size);

You can find more examples on their web-site.