CODEWITHSUNDEEP
cdictionarycompressionzlibdeflate
otrevor
otrevor

Reputation: 21

cannot compress stream of data setting a dictionary with zlib

I have been attempting to set a dictionary in zlib by modifying the zpipe.c example. Namely, I have a file with 32768 characters that I want to turn into a dictionary. So I modified zpipe (http://www.zlib.net/zpipe.c).

On the def() function, I added:

char dict[32768];
FILE *fd = fopen("dictB.txt", "r");
ssize_t test = fread(dict, 32768, 1, fd);
int lenDict = (int) sizeof(dict);
fclose(fd);

After deflateInit(), I have added the following

ret = deflateSetDictionary(&strm, (const Bytef*) dict, lenDict);

For good measure, I added the deflateSetDictionary and every point before calling deflate()

On the inf() function, I added the same dictionary (repeating for completeness): 

char dict[32768];
FILE *fd = fopen("dictB.txt", "r");
ssize_t test = fread(dict, 32768, 1, fd);
int lenDict = (int) sizeof(dict);
fclose(fd);

and after the inflate() call, I modified zpipe.c such that it could accept a dictionary call:

ret = inflate(&strm, Z_NO_FLUSH);
if (ret==Z_NEED_DICT){
   ret = inflateSetDictionary(&strm, (const Bytef*) dict, lenDict); 
}
assert(ret != Z_STREAM_ERROR);  /* state not clobbered */
switch (ret) {
 case Z_DATA_ERROR:
 case Z_MEM_ERROR:
    (void)inflateEnd(&strm);
    return ret;
}

Now, after running the compression 

$ ./zpipe < file.txt > file.gz

Then everything runs without errors

But when I try to decompress

$ ./zpipe -d < file.gz > file.dec.txt

Then I obtain an error related to Z_DATA_ERROR:

zpipe: invalid or incomplete deflate data

This error does not show up when implementing the deflateSetDictionary calls. I am aware that this error is related to deflateSetDictionary, and perhaps when implemented using buffers, since there is no error when running other examples with dictionaries (such as http://www.chuntey.com/Source/Zlib/example.c)

Upvotes: 2

Views: 632

Answers (1)

Armali
Armali

Reputation: 19375

After the inflateSetDictionary() you need to run inflate() again. Otherwise you'll fall out of the inner loop and overwrite the input that was read. – Mark Adler

Upvotes: 1

Related Questions