CODEWITHSUNDEEP
c++compressionlz4
mambo_sun
mambo_sun

Reputation: 539

How to decompress less than original size with Lz4 library?

I'm using LZ4 library and when decompressing data with:

int LZ4_decompress_fast_continue (void* LZ4_streamDecode, const char* source, char* dest, int originalSize);

I need only first n bytes of the originally encoded N bytes, where n < N. So in order to improve the performance, it makes sense to decompress only a part of the original buffer.

I wonder if I can pass n instead of N to the originalSize argument of the function?

My initial test showed, that it's not possible (I got incorrectly decompressed data). Though maybe there is a way, for example if n is a multiple of some CHUNK_SIZE? All original N bytes were compressed with 1 call of a compress function.

Upvotes: 2

Views: 2314

Answers (2)

zbyszek
zbyszek

Reputation: 5305

This seems to have been implemented in lz4 1.8.3.

Upvotes: 1

Cyan
Cyan

Reputation: 13968

LZ4_decompress_safe_continue() and LZ4_decompress_fast_continue() can only decode full blocks. They consider a partial block as an error, and report it as such. They also consider that if there is not enough room to decompress a full block, it's also an error.

The functionality you are looking for doesn't exist yet. But there is a close cousin that might help.

LZ4_decompress_safe_partial() can decode a part of a block. Note that, in contrast with _continue() variants, it only works on independent blocks. Note also that the compressed block must nonetheless be complete, and the output buffer must nonetheless have enough space to decode the entire block. So the only advantage provided by this function is speed : if you want only the first 10 bytes, it will stop as soon as it has generated enough bytes.

"as soon as" doesn't mean "exactly at 10". It could be much later, and in the worst case, it could be after decoding the entire block. That's because the internal decoding engine is still the same : it decodes entire sequences, and doesn't "break them" in the middle, for speed considerations.

If you need to extract less bytes than a full block in order to save some memory, I'm afraid there is no solution yet. Report it as a feature request to upstream.

Upvotes: 4

Related Questions