CODEWITHSUNDEEP
amazon-s3
Austin Huang
Austin Huang

Reputation: 131

How to get a pre-signed URL that downloads file with http compression

Here is my code in node.js:

    const downloadURL = await s3.getSignedUrlPromise('getObject', {
      Bucket: BUCKET_NAME,
      Key: 'key to a large json file',
    });

One got the URL, I want to download a very large JSON file stored in S3 from browser. Since it is large, I would like to use HTTP compression which would compress a 20MB JSON to less than 1MB. I could not find anywhere how to do it or whether it is at all possible with S3 APIs.

I also tried to do below when using the signed URL to download file and it seems not work.

    const dataRes = await fetch(downloadURL, {
      headers: {
        'Accept-Encoding': 'gzip, deflate',
      },
      method: 'GET',
    });

Hope somebody could help me out. Thanks a lot!

Upvotes: 5

Views: 3417

Answers (1)

Austin Huang
Austin Huang

Reputation: 131

After doing some study, I have resolved this. Post here and hope it is helpful to others.

  • You cannot ask S3 to compress file on the fly when getObject or using signed URL to getObject
  • You would have to save the zipped file into S3 in the first place. In Linux, using below command to do it: 
    gzip -9 <file to compress>
  • Upload the zipped file to S3
  • Use below code to generate the signed URL: 
        const downloadURL = await s3.getSignedUrlPromise('getObject', {
      Bucket: BUCKET_NAME,
      Key: 'key to a large zipped json file',
      ResponseContentEncoding: 'gzip',
      ResponseContentType: 'application/json',
    });
  • Use below code to download from the signed URL: 
      const res = await fetch(downloadurl);
  const jsonData = await res.json();

Upvotes: 8

Related Questions