CODEWITHSUNDEEP
javacompressionapache-commons
Tej Kiran
Tej Kiran

Reputation: 2228

How to make compression more faster in java to archiving file

I am working on java application, where I have to manage the files into tar/zip format. So I am creating tar/zip from csv file. Now I am looking for the way, by which I can do it faster. Because I have to run this for millions of files.

I am using following code and what to make it more faster. Following Libs are Using...

FileUtils of org.apache.commons.io

TarArchiveEntry of org.apache.commons.compress.archivers.tar

try {
        // Create staging file output stream
        File temp = new File(getFilePath(objectm));
        log.debug("temping " + objectm.getPath());
        outputStream = new FileOutputStream(temp);

        // Create GZip pass-thru stream
        if (isCompressionEnabled) {
            compressionStream = new
                    CompressionStream(outputStream, getCompressionLevel(objectm));
        }

        // Create MD5 hash
        final MessageDigest outputDigest = MessageDigest.getInstance("MD5");
        md5OutputStream = new DigestOutputStream(isCompressionEnabled ? compressionStream : outputStream, outputDigest);

        // Create tar stream
        tarStream = new TarArchiveOutputStream(new BufferedOutputStream(md5OutputStream));
        tarStream.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);
        tarStream.setBigNumberMode(TarArchiveOutputStream.BIGNUMBER_STAR);

        // tar the first object
        TarArchiveEntry entry = new TarArchiveEntry(objectm.getHierarchy());
        entry.setSize(objectm.getOriginalSize());
        entry.setModTime(objectm.getLastModified().getMillis());
        tarStream.putArchiveEntry(entry);
        org.apache.commons.io.IOUtils.copyLarge(inputStream, tarStream);


        // Collect properties to return
        String digest = Hex.encodeHexString(outputDigest.digest());
        objectm.setChecksum(digest);
        objectm.setDate(DateTime.now());
        objectm.setCompressSize(FileUtils.sizeOf(temp));          
        tarStream.finish();
        log.debug("Completed.");

    } catch (Exception e) {
        throw new Exception("Exception: Creating tar" , e);
    } finally {
        org.apache.cobjectmmons.io.IOUtils.closeQuietly(inputStream);
        org.apache.cobjectmmons.io.IOUtils.closeQuietly(tarStream);
        org.apache.cobjectmmons.io.IOUtils.closeQuietly(cobjectmpressionStream);
        org.apache.cobjectmmons.io.IOUtils.closeQuietly(md5OutputStream);
        org.apache.cobjectmmons.io.IOUtils.closeQuietly(outputStream);
    }

Here in second method we are moving the object from temp to actual location.

try {
        File src = new File(getFilePath(objectm));
        File dst = new File(sDestinationFile);
        FileUtils.moveFile(src, dst);
        boolean readableFlag = dst.setReadOnly();
    } catch (IOException e) {
        throw new Exception("Unable to move to destination.", e);
    }

Upvotes: 0

Views: 2772

Answers (1)

Anil
Anil

Reputation: 1765

Use LZ4, that is currently the fastest algorithm for compressing files. Code sample below.

LZ4Factory factory = LZ4Factory.fastestInstance();

byte[] data = "12345345234572".getBytes("UTF-8");
final int decompressedLength = data.length;

// compress data
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
byte[] compressed = new byte[maxCompressedLength];
int compressedLength = compressor.compress(data, 0, decompressedLength, compressed, 0, maxCompressedLength);

// decompress data
// - method 1: when the decompressed length is known
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[decompressedLength];
int compressedLength2 = decompressor.decompress(compressed, 0, restored, 0, decompressedLength);
// compressedLength == compressedLength2

// - method 2: when the compressed length is known (a little slower)
// the destination buffer needs to be over-sized
LZ4SafeDecompressor decompressor2 = factory.safeDecompressor();
int decompressedLength2 = decompressor2.decompress(compressed, 0, compressedLength, restored, 0);
// decompressedLength == decompressedLength2

Reference taken from https://github.com/jpountz/lz4-java

Upvotes: 1

Related Questions