CODEWITHSUNDEEP
compressionzlibdeflategzip
euccas
euccas

Reputation: 775

zlib deflate: why does it accumulate short data and doesn't start compression until input buffer is full?

In zlib's deflate implementation, noticed when the given data has small length, zlib function will copy the data to input buffer, and only start compression when input buffer is full.

When the given data's length is longer than input buffer size, zlib function starts compression directly.

The default input buffer size is 8KB.

I'm wondering about the benefits of accumulating short data into the input buffer. What I can think of are:

Could anyone who knows more insights share your thoughts with me, or point to me some references?

Related code:

if (len < state->size) {
    /* copy to input buffer, compress when full */
    do {
        if (strm->avail_in == 0)
            strm->next_in = state->in;
        n = state->size - strm->avail_in;
        if (n > len)
            n = len;
        memcpy(strm->next_in + strm->avail_in, buf, n);
        strm->avail_in += n;
        state->x.pos += n;
        buf = (char *)buf + n;
        len -= n;
        if (len && gz_comp(state, Z_NO_FLUSH) == -1)
            return 0;
    } while (len);
}
else {
    /* consume whatever's left in the input buffer */
    if (strm->avail_in && gz_comp(state, Z_NO_FLUSH) == -1)
        return 0;

    /* directly compress user buffer to file */
    strm->avail_in = len;
    strm->next_in = (voidp)buf;
    state->x.pos += len;
    if (gz_comp(state, Z_NO_FLUSH) == -1)
        return 0;
}

Upvotes: 0

Views: 980

Answers (1)

Mark Adler
Mark Adler

Reputation: 112239

The deflate compressed data format is composed of blocks, which have a header that depends on the block data. Therefore the output of deflate comes a block at a time, with nothing written (except a zlib or gzip header) until the first block is completed.

zlib's deflate accumulates data until, for the default settings, 16K symbols have been generated. A symbol is either a single byte, coded as a literal, or a length/distance pair, which codes a copy of up to 258 bytes somewhere in the preceding 32K of uncompressed data. So you will accumulate from 16K to as much as 4MB of uncompressed data (for highly compressible data) before the first block is emitted.

Once that data is accumulated, zlib decides what kind of block to construct, and then does so, creating the header, which for a dynamic block describes the Huffman codes in the block, and then creates the coded symbols for that block. Or it creates a stored or static block, whatever results in the fewest number of bits. Only then is that compressed data available.

This is repeated, so the deflate compressed data will appear to become available in bursts, if you are feeding it small input buffers. It's doing exactly the same thing with large input buffers, but then you're just not seeing the latency as readily, but it's still there.

Upvotes: 3

Related Questions