CODEWITHSUNDEEP
node.jszlib
Thusitha Nuwan
Thusitha Nuwan

Reputation: 611

Incorrect Header Check when using zlib in node.js

I am trying to send a simple HTTP POST request, retrieve the response body.Following is my code. I am getting

Error: Incorrect header check

inside the "zlib.gunzip" method. I am new to node.js and I appreciate any help.

;

    fireRequest: function() {

    var rBody = '';
    var resBody = '';
    var contentLength;

    var options = {
        'encoding' : 'utf-8'
    };

    rBody = fSystem.readFileSync('resources/im.json', options);

    console.log('Loaded data from im.json ' + rBody);

    contentLength = Buffer.byteLength(rBody, 'utf-8');

    console.log('Byte length of the request body ' + contentLength);

    var httpOptions = {
        hostname : 'abc.com',
        path : '/path',
        method : 'POST',
        headers : {
            'Authorization' : 'Basic VHJhZasfasNWEWFScsdfsNCdXllcjE6dHJhZGVjYXJk',
            'Content-Type' : 'application/json; charset=UTF=8',
            // 'Accept' : '*/*',
            'Accept-Encoding' : 'gzip,deflate,sdch',
            'Content-Length' : contentLength
        }
    };

    var postRequest = http.request(httpOptions, function(response) {

        var chunks = '';
        console.log('Response received');
        console.log('STATUS: ' + response.statusCode);
        console.log('HEADERS: ' + JSON.stringify(response.headers));
        // response.setEncoding('utf8');
        response.setEncoding(null);
        response.on('data', function(res) {
            chunks += res;
        });

        response.on('end', function() {
            var encoding = response.headers['content-encoding'];
            if (encoding == 'gzip') {

                zlib.gunzip(chunks, function(err, decoded) {

                    if (err)
                        throw err;

                    console.log('Decoded data: ' + decoded);
                });
            }
        });

    });

    postRequest.on('error', function(e) {
        console.log('Error occured' + e);
    });

    postRequest.write(rBody);
    postRequest.end();

}

Upvotes: 49

Views: 84597

Answers (4)

OfirD
OfirD

Reputation: 10460

In addition to @Nitzan Shaked's answer, this might be related to the Node.js version.

What I experienced is that https.request (OP uses http.request, but it may behave the same) already decompresses the data under the hood, so that once you accumulate the chunks into a buffer, all you're left with is to call buffer.toString() (assuming utf8 as an example). I myself experienced it in this other answer and it seems to be related to Node.js version.

I'll end up this answer with a live demo of a similar working code which may come handy for future readers (it queries StackExchange API, gets a gzip compressed chunks, and then decompress it):

  • It includes a code that works on 14.16.0 (current StackBlitz version) - which, as I described, already decompresses the data under the hood - but not on Node.js 15.13.0,
  • It includes a commented-out code that works for Node.js 15.13.0 the latter but not for 14.16.0.

Upvotes: 1

arnaudjnn
arnaudjnn

Reputation: 1063

I had this error trying to loop with fs.readdirSync but there is a .Dstore file so the unzip function was applied to it.

Be careful to pass only .zip/gz

import gunzip from 'gunzip-file';

const unzipAll = async () => {
  try {
    const compFiles = fs.readdirSync('tmp')
    await Promise.all(compFiles.map( async file => {
      if(file.endsWith(".gz")){
        gunzip(`tmp/${file}`, `tmp/${file.slice(0, -3)}`)
      }
    }));
  }
  catch(err) {
    console.log(err)
  }
}

Upvotes: 1

Roman
Roman

Reputation: 21785

In our case we added 'accept-encoding': 'gzip,deflate' to the headers and code started working (solution credited to Arul Mani):

var httpOptions = {
    hostname : 'abc.com',
    path : '/path',
    method : 'POST',
    headers : {
        ...
        'accept-encoding': 'gzip,deflate'
    }
};

Upvotes: 2

Nitzan Shaked
Nitzan Shaked

Reputation: 13598

response.on('data', ...) can accept a Buffer, not just plain strings. When concatenating you are converting to string incorrectly, and then later can't gunzip. You have 2 options:

1) Collect all the buffers in an array, and in the end event concatentate them using Buffer.concat(). Then call gunzip on the result.

2) Use .pipe() and pipe the response to a gunzip object, piping the output of that to either a file stream or a string/buffer string if you want the result in memory.

Both options (1) and (2) are discussed here: http://nickfishman.com/post/49533681471/nodejs-http-requests-with-gzip-deflate-compression

Upvotes: 27

Related Questions