CODEWITHSUNDEEP
javascriptnode.jselectron
Polar
Polar

Reputation: 3547

JavaScript how can I make sure the downloaded file is successful and not corrupted

I'm playing around with Electron-Node.js and I'm trying to download a file from a server and save it into the local disk.

The code below works as expected, though I'm not sure if this is the best method to do it.

<script>
    const { ipcRenderer } = require('electron');
    var fs = require('fs');

    // (2) convert the blob to file.
    async function saveBlobToFile(blob, filename) {
        let fileData = new Int8Array(await blob.arrayBuffer());
        fs.writeFileSync(filename, fileData);
        console.log("Blob successfully saved to file.");
        //is the file healthy though?
    }

    // (1) (download the file)
    function download(downloadURL, downloadDirectory){
        var filename = "test.zip"; //this could be exe, images, pdf
        if (!fs.existsSync(downloadDirectory)){ 
            fs.mkdirSync(downloadDirectory, { recursive: true });
            console.log("Download Directory doesn't exist, created.");
        }
        console.log("Downloading...");
        fetch(downloadURL)
            .then(response => response.blob())
            .then(blob => {
                console.log("Downloaded and is now saving...");
                saveBlobToFile(blob, downloadDirectory+"\\"+filename);
            }).catch(console.error);
    }
        
    download("https://sample.com/test.zip", "SOME_DIRECTORY_HERE"); //let's try to download the file.
</script>

Here comes the WhatIF;

I test it with a file size of 50MB and it works, the file is healthy. But what if the file size is more than 200MB or let's say a large file and the user has only a very limited internet connection? The file might get corrupted, how to avoid this? How can we check or ensure that the file is successfully downloaded and is not corrupted?

I'm thinking, maybe I can check the MD5 Checksum of the downloaded file and compare it to the file from the Server. But is this enough?

If the MD5 Checksum is different from the Server, then redownload the file and recheck it, but again this will be a waste of data for a user with a very limited internet connection.

Upvotes: 0

Views: 638

Answers (1)

Tisamu
Tisamu

Reputation: 154

As marc said, there is no common reason that your file get corrupted

But

I'm thinking, maybe I can check the MD5 Checksum of the downloaded file and compare it to the file from the Server. But is this enough?

is for me the write answer to your own question.

Then

If the MD5 Checksum is different from the Server, then redownload the file and recheck it, but again this will be a waste of data for a user with a very limited internet connection.

Well, you can split your file in chunks.

If you have large files to transfer (> 1Gb for exemple), you could split it in chunks of 100Mb and transfer them one by one, and check checksums individually, then concatenate them and check the full checksum again.

eg:

  1. Request download to the server, the server respond by giving the number of files or the size of the final file
  2. On the first file download request, split the 1Gb file into 10 files in a tmp folder on the server.
  3. After each download, check the checksum of this single part.
  • If wrong checksum, only re-download this part
  1. After all the parts has been download, concatenate them (Probably with Buffer.from(fileParts) and validate the final checksum

It's a just a quick thought about a potential solution to your problem.

Hopes it helps you.

Upvotes: 1

Related Questions