CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
WebDevGuy2
WebDevGuy2

Reputation: 1249

Retrieving the progress of getObject (aws-sdk)

I'm using node.js with the aws-sdk (for S3).... When I am downloading a huge file from s3, how can I regularly retrieve the progress of the download so that the front-end can show a progress bar? Currently I am using getObject. (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property)

The code to download the file works. Here's a snippet of my code...

 return await new Promise((resolve, reject) => {
  this.s3.getObject(params, (error, data) => {
    if (error) {
      reject(error);
    } else {
      resolve(data.Body);
    }
  });

I'm just not sure how to hook into the progress as it's downloading. Thanks in advance for any insight!

Upvotes: 3

Views: 2549

Answers (2)

Avocado
Avocado

Reputation: 901

STREAMING OPTION

Another option is to use a stream and track the amount of bytes received:

    const { ContentLength: contentLength } = await s3.headObject(params).promise();

    const rs = s3.getObject(s3Params).createReadStream();

    let progress = 0;
    rs.on('data', function (chunk) {
      // Advance your progress by chunk.length
      progress += chunk.length;
      console.log(`Progress: ${progress / contentLength}%`);
    });

    // ... pipe to write stream

Upvotes: 0

Allan Chua
Allan Chua

Reputation: 10185

You can utilize S3 byte-range fetching which allows the fetching of small parts of a file in S3. This capability then allows us to fetch large objects by dividing the file download into multiple parts which brings the following advantages:

  • Part download failure does not require full re-downloading of the file.
  • Download pause/resume capability.
  • Download progress tracking
  • Retry packets that failed or interrupted by network issues
  • Sniff headers located in the first few bytes of the file if we just need to get metadata from the files.

You can split the file download by your size of choice (I propose 1-4mb at a time) and download the parts chunk by chunk, when each of the get object promises complete, you can trace how many have completed. A good start is by looking at the AWS documentation.

Upvotes: 3

Related Questions