CODEWITHSUNDEEP
jsonnode.jszlibsha
Micah Henning
Micah Henning

Reputation: 2185

SHA Hashes Don't Match between Node Crypto and macOS sha256sum

I'm fetching a large, gzipped JSON file from an Internet source in node.js. That source also provides a meta file that contains a SHA256 hash. If I write the file to disk and sum the resulting file, the hashes match; however, when summing the buffer in NodeJS, the hashes do not match.

const https = require('request-promise-native');
const request = require('request');
const zlib = require('zlib');
const crypto = require('crypto');

const getList = async (list) => {
  // Start by getting meta file to SHA result
  const meta = await https(`https://example.com/${list}.meta`);
  const metaHash = meta.match(/sha256:(.+)$/im)[1].toLowerCase();

  // metaHash = "f36c4c75f1293b3d3415145d78a1ffc1b8b063b083f9854e471a3888f77353e1"

  // Download and unzip whole file
  const chunks = [];
  const file = await new Promise((resolve, reject) => {
    const stream = request(`https://example.com/${list}.json.gz`);

    stream.on('data', chunk => chunks.push(chunk));
    stream.on('error', reject);
    stream.on('end', () => {
      const buffer = Buffer.concat(chunks);

      // Unzip
      zlib.gunzip(buffer, (error, unBuffer) => {
        // TEST:  Write to disk
        fs.writeFile('/tmp/test.json', unBuffer);

        // Check SHA hash
        const afterHash = crypto.createHash('sha256');
        afterHash.update(unBuffer);
        const hash = afterHash.digest('hex');

        // hash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

        // metaHash =/= hash
        if (metaHash === hash) resolve(unBuffer.toString());
        else {
          // reject(`SHA hashes do not match for ${list}`);
          console.log(`${list}\n${metaHash}\n${hash}`);
        }
      });
    });
  });
};

But from my macOS terminal, it matches:

$ sha256sum /tmp/test.json
f36c4c75f1293b3d3415145d78a1ffc1b8b063b083f9854e471a3888f77353e1  /tmp/test.json

That leads me to believe the file was downloaded and unzipped correctly. Did I implement node.js crypto incorrectly? Am I doing something else wrong? Thanks!

UPDATE

I just realized I'm getting the same hash e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 from crypto for every file I try, so I'm definitely doing something wrong here...

Upvotes: 0

Views: 558

Answers (2)

Micah Henning
Micah Henning

Reputation: 2185

I never did figure out what I was doing wrong. I used hasha instead, which solved the issue.

const hasha = require('hasha');

...

  // Unzip
  zlib.gunzip(buffer, (error, unBuffer) => {
    // Check SHA hash
    const hash = hasha(unBuffer, { algorithm: 'sha256' });

    if (metaHash === hash) resolve(unBuffer.toString());
    else reject(`SHA hashes do not match for ${list}`);
  });

Upvotes: 0

Mark Adler
Mark Adler

Reputation: 112512

Your e3b0c... is the SHA-256 of the empty sequence, i.e. zero input bytes.

Upvotes: 2

Related Questions