CODEWITHSUNDEEP
node.jshttpsgzipzlib
Robert Oschler
Robert Oschler

Reputation: 14375

Incorect header check error from Zlib.gunzip() in Node.JS app using the HTTPS module

I have a Node.JS app and I am using the https module to make GET requests to a web server. The headers in the response coming back have the content-type set to gzip. I have directly inspected the data and it does appear to be compressed data and definitely not plain text.

I accumulate the chunks as they come in. I then try decompressing the accumulated data using zlib. So far everything I tried results in an "Incorrect header check" error when execute the decompression call. The code below shows the use of a Buffer object with type set to binary. I previously tried passing the accumulated data directly to the decompression call but that failed too.

Why doesn't this work?

// Make the request to the designated external server.
const httpsRequest = https.request(postOptions,
    function(extRequest)
    {
        console.log('(httpsRequest) In request handler.');
        
        // Process the response from the external server.
        let dataBody = "";
        
        // The data may come to us in pieces.  The 'on' event handler will accumulate them for us.
        let iNumSlices = 0;
        extRequest.on('data', function(dataSlice) {
            iNumSlices++;
            console.log('(httpsRequest:on) Received slice # ' + iNumSlices +'.');
            dataBody += dataSlice;
        });
        
        //  When we have received all the data from the external server, finish the request.
        extRequest.on('end', function() {
            // SUCCESS: Return the result to AWS.
            console.log('(httpsRequest:end) Success.  Data body length: ' + dataBody.length +'.');
            console.log('(httpsRequest:end) Content: ');
            
            let buffer = Buffer.from(dataBody, "binary");
            
            // Check for GZip compressed data.
            if (extRequest.headers['content-encoding'] == 'gzip') {
                // Decompress the data.
            
                zlib.gunzip(buffer, (err, buffer) => {
                    if (err) {
                        // Reject the promise with the error.
                        reject(err);
                        return;
                    } else {
                        console.log(errPrefix + buffer.toString('utf8'));
                    }
                });
            } else {
                console.log(errPrefix + dataBody);
                let parsedDataBodyObj = JSON.parse(dataBody);
                resolve(parsedDataBodyObj);
            }
        });
    });

Upvotes: 0

Views: 1325

Answers (1)

OfirD
OfirD

Reputation: 10460

  • You may have it in you actual code - but the code snippet doesn't include a call to end(), which is mandatory.

  • It may be related to the way you accumulate the chunks with dataBody += dataSlice.

    Since the data is compressed, this (probably) means that the type of a chunk is already a Buffer, and using += to concatenate it into a string seems to mess it up, even though you later call Buffer.from.

    Try replacing it with making dataBody an empty array, then push chunks into it, then finally call Buffer.concat(dataBody).

  • Another options is that https.request already decompresses the data under the hood, so that once you accumulate the chunks into a buffer (as detailed in the previous section), all you're left with is to call buffer.toString(). I myself experienced it in this other answer and it seems to be related to Node.js version.

I'll end up this answer with a live demo of a similar working code which may come handy for you (it queries StackExchange API, gets a gzip compressed chunks, and then decompress it):

  • It includes a code that works on 14.16.0 (current StackBlitz version) - which, as I described, already decompresses the data under the hood - but not on Node.js 15.13.0,
  • It includes a commented-out code that works for Node.js 15.13.0 the latter but not for 14.16.0.

Upvotes: 1

Related Questions