CODEWITHSUNDEEP
c++compressionzip
SinisterMJ
SinisterMJ

Reputation: 3509

Compression library for C / C++ able to deal with more than 32 bit elements in the array

I have a problem in the fact that I need to compress around a 6 GB std::vector() (1.5 billion floats in it), and up to now I have used lz4, but it only handles int count of chars. Since I have 6 billion chars in my vector, that would need 33bit to represent, and the compression with LZ4 does not work as I need it to.

From what I saw at the zlib libraries, it takes int as well as input for the length of the to compressed data.

Do I need to segment my data, or is there a framework around able to deal with more than 32bit of chars, or am I missing something?

Upvotes: 4

Views: 682

Answers (2)

vonbrand
vonbrand

Reputation: 11791

Take a look at XZ, it seems to handle really big sizes. The CLI executables themselves are thin wrappers over a library, so this should fit your bill.

OTOH, a stream of binary floats shouldn't compress that well...

Upvotes: 0

user149341
user149341

Reputation:

Use zlib, and pass the array in as several chunks. The DEFLATE algorithm used by zlib has a window size of about 32 KB, and it already buffers the compressed data, so passing the data in as multiple chunks will not affect the compression efficiency.

Upvotes: 2

Related Questions