How to compress/decompress buffer using Fast-LZMA2

Question

I want to compress/decompress a unsigned char buffer using fast-LZMA2 by 7Zip : https://github.com/conor42/fast-lzma2

In the sample there's two function :

static int compress_file(FL2_CStream *fcs)
{
    unsigned char in_buffer[8 * 1024];
    unsigned char out_buffer[4 * 1024];
    FL2_inBuffer in_buf = { in_buffer, sizeof(in_buffer), sizeof(in_buffer) };
    FL2_outBuffer out_buf = { out_buffer, sizeof(out_buffer), 0 };
    size_t res = 0;
    size_t in_size = 0;
    size_t out_size = 0;
    do {
        if (in_buf.pos == in_buf.size) {
            in_buf.size = fread(in_buffer, 1, sizeof(in_buffer), fin);
            in_size += in_buf.size;
            in_buf.pos = 0;
        }
        res = FL2_compressStream(fcs, &out_buf, &in_buf);
        if (FL2_isError(res))
            goto error_out;

        fwrite(out_buf.dst, 1, out_buf.pos, fout);
        out_size += out_buf.pos;
        out_buf.pos = 0;

    } while (in_buf.size == sizeof(in_buffer));
    do {
        res = FL2_endStream(fcs, &out_buf);
        if (FL2_isError(res))
            goto error_out;

        fwrite(out_buf.dst, 1, out_buf.pos, fout);
        out_size += out_buf.pos;
        out_buf.pos = 0;
    } while (res);
    fprintf(stdout, "\t%ld -> %ld\n", in_size, out_size);
    
    return 0;

error_out:
    fprintf(stderr, "Error: %s\n", FL2_getErrorName(res));
    return 1;
}
static int decompress_file(FL2_DStream *fds)
{
    unsigned char in_buffer[4 * 1024];
    unsigned char out_buffer[8 * 1024];
    FL2_inBuffer in_buf = { in_buffer, sizeof(in_buffer), sizeof(in_buffer) };
    FL2_outBuffer out_buf = { out_buffer, sizeof(out_buffer), 0 };
    size_t res;
    size_t in_size = 0;
    size_t out_size = 0;
    do {
        if (in_buf.pos == in_buf.size) {
            in_buf.size = fread(in_buffer, 1, sizeof(in_buffer), fout);
            in_size += in_buf.size;
            in_buf.pos = 0;
        }
        res = FL2_decompressStream(fds, &out_buf, &in_buf);
        if (FL2_isError(res))
            goto error_out;
        /* Discard the output. XXhash will verify the integrity. */
        out_size += out_buf.pos;
        out_buf.pos = 0;
    } while (res && in_buf.size);
    
    fprintf(stdout, "\t%ld -> %ld\n", in_size, out_size);
    
    return 0;

error_out:
    fprintf(stderr, "Error: %s\n", FL2_getErrorName(res));
    return 1;
}

But I have no idea how to make it work with a buffer and also without size limit like 8*1024 like zlib deflate compression.

I want something like LZMA2_Compress(void* buffer,size_t bufferSize);

and LZMA2_Decompress(void* buffer,size_t bufferSize);

I want to use this algorithm on some heavy files and Fast LZMA2 is the fastest high ratio compression I found, Please don't suggest me using other methods.

Here's my test code, It's working but just need to correct information: https://gist.github.com/Bit00009/3241bb66301f8aaba16074537d094e61

Answer 1

Check the header file for all of the functions available. This one looks like the one you need. You will need to cast your buffers as (void *).

High level functions

fast-lzma2.h

...
/*! FL2_compress() :
 *  Compresses `src` content as a single LZMA2 compressed stream into already allocated `dst`.
 *  Call FL2_compressMt() to use > 1 thread. Specify nbThreads = 0 to use all cores.
 *  @return : compressed size written into `dst` (<= `dstCapacity),
 *            or an error code if it fails (which can be tested using FL2_isError()). */
FL2LIB_API size_t FL2LIB_CALL FL2_compress(void* dst, size_t dstCapacity,
    const void* src, size_t srcSize,
    int compressionLevel);
...

Management of memory and options

To do explicit memory management (set dictionary size, buffer size, etc.) you need to create a context:

fast-lzma2.h

/*= Compression context
 *  When compressing many times, it is recommended to allocate a context just once,
 *  and re-use it for each successive compression operation. This will make workload
 *  friendlier for system's memory. The context may not use the number of threads requested
 *  if the library is compiled for single-threaded compression or nbThreads > FL2_MAXTHREADS.
 *  Call FL2_getCCtxThreadCount to obtain the actual number allocated. */
typedef struct FL2_CCtx_s FL2_CCtx;
FL2LIB_API FL2_CCtx* FL2LIB_CALL FL2_createCCtx(void);

than you can use FL2_CCtx_setParameter() to set the parameters in the context. The possible values for the paramters are listed in FL2_cParameter , and the value FL2_p_dictionarySize will allow you to set the dictionary size.

/*! FL2_CCtx_setParameter() :
 *  Set one compression parameter, selected by enum FL2_cParameter.
 *  @result : informational value (typically, the one being set, possibly corrected),
 *            or an error code (which can be tested with FL2_isError()). */
FL2LIB_API size_t FL2LIB_CALL FL2_CCtx_setParameter(FL2_CCtx* cctx, FL2_cParameter param, size_t value);

Finally you can compress the buffer by calling FL2_compressCCtx()

/*! FL2_compressCCtx() :
 *  Same as FL2_compress(), but requires an allocated FL2_CCtx (see FL2_createCCtx()). */
FL2LIB_API size_t FL2LIB_CALL FL2_compressCCtx(FL2_CCtx* cctx,
    void* dst, size_t dstCapacity,
    const void* src, size_t srcSize,
    int compressionLevel);