CODEWITHSUNDEEP
node.jsamazon-s3md5checksummd5sum
Calvin
Calvin

Reputation: 417

checksum md5 value generated does not match

I am using md5sum on my backend to calculate md5 checksum of files I'm uploading to S3, and I have lambda in node js crypto module to stream the content and calculate md5 with the specified byte range of s3 parameter.

However, when I stream the whole file the md5 match with what I have calculated with md5sum tool, but when I specify same byte-range on both md5sum and s3 Range parameter the resulting md5 are different?

Does anyone can explain why this is? any help is appreciated. thanks!

Here's the md5 calculation from lambda:

let s3params = {
    Bucket: 'bucket',
    Key: filename.toString(),
    Range: "bytes=0-1073741824"
};
let hash = crypto.createHash('md5');
let stream = s3.getObject(s3params).createReadStream();
stream.on('data', (data)=>{
    hash.update(data);
}
);

stream.on('end', ()=>{
    var digest = hash.digest('hex');
    console.log("this is md5 value from digest: " + digest);
    callback(null, digest);
    digest = digest.toString().replace(/[^A-Za-z 0-9 \.,\?""!@#\$%\^&\*\(\)-_=\+;:<>\/\\\|\}\{\[\]`~]*/g, '');
});

output from lambda md5 is 75cbf7ae988a9086b1faa780c4e4bdc2

here's the md5 calculation from md5sum in my bash script:

bigchecksum=$(head -c 1073741824 ${file} | md5sum | cut -d ' ' -f 1)

and the output is d61b42882135eebb449fc3904dc708b2

Upvotes: 0

Views: 1245

Answers (1)

Calvin
Calvin

Reputation: 417

figured out the issue, the byte range on S3 parameter header should be 0-1073741823 instead of 0-1073741824, then the md5 matches md5sum output

Upvotes: 0

Related Questions