Reputation: 2836
I know how big I expect the decompressed data to be, what I want to know is how many bytes I will have to read from the block of memory containing the compressed data before I reach this size.
LZ4_decompress_fast did this, exactly, but has been deprecated in newer versions, so was originally thinking I could just replace it with calls to LZ4_decompress_safe, but the latter function returns the size of the decompressed buffer, not the count of compressed bytes that were read to get to the expected size as LZ4_decompress_fast did. I was thinking that maybe I need to do some sort of streaming decompression, and keep on decompressing until I reach the right size (or until I get some kind of inconsistent data on the source buffer that does not correspond with compressed data), and writing a function that would basically act like a plugin replacement for LZ4_decompress_fast, except using API calls that are not deprecated. But I can't find any docs (maybe my google-fu is not strong, I don't know) that explain clearly how to do this, and in particular how to figure out how many bytes were read from the source buffer to expand into the destination to reach the expected size (or until bytes are being read from the source that don't make sense for compressed data, and attempting to decompress more byte from the source buffer would fail).
Regrettably, most of the examples I was able to find with google on using it are oriented to very specific use cases, and I cannot figure out how to generalize any of these to get the information that I actually need.
Any assistance at all would be greatly appreciated, or pointers to documentation that might be better able to help me figure out how do to this would also be welcome.
EDIT:
While the pointers to related questions are usually helpful, none of the ones listed actually explain how to figure out how many bytes were decompressed from the source buffer. Many of them have nothing to do with the LZ4 api.
I figured I would just have to repeatedly call LZ4_decompress_safe_continue (which appears to decompress only a single block) until I get an error, advancing through the source buffer as I go, but that doesn't tell me how many bytes were actually in the compressed block, and in turn where to advance the pointer to for the next call. Essentially, I need a function that can tell me how many bytes from the src will get read in a single call to LZ4_decompress_safe_continue, so I can know how much I need to advance the pointer by for the next call. I can avoid the aforementioned error by terminating my loop if the value is 0 (or otherwise too small to hold any data), effectively obtaining both how many compressed bytes were in the buffer, and how many bytes it was decompressed into.
The reason that I need to know this is that my application originally called lz4_compress_default on some data, and then immediately after the compressed data, put some application specific data into the buffer, and I only ever saved the combined size of both the compressed size and the application data.
Now I need to decompress what was in there, and figure out what the offset of my application specific data is in the source buffer. Yes, it was probably short-sighted to not be saving the size of the compressed data separately in the first place, but that's water under the bridge now. I need a binary compatible way to read that data back and figure out where the application specific data starts in the buffer.
Upvotes: 0
Views: 98