CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-s3browserupload
Felipe
Felipe

Reputation: 153

How can I safely upload a large file from the browser to a bucket on Amazon S3?

I need to allow a user to upload a large file (over 2GB) from the web application and I don't want to expose the company's access keys.

One of my teammates found that to upload without the keys we need to use presigned urls. In this way, the frontend requests a presigned url to the backend and, after receiving it, the frontend uploads the file via an HTTP request. Following is the code that does this:

const presignedUrl = '...'

const body = {
    data: file,
};

const req = new HttpRequest('PUT', presignedUrl, body, {
    headers: new HttpHeaders({
        'Content-Type': contentType,
        'x-amz-acl': 'public-read',
    }),
    reportProgress: true,
});

I found that to upload large files via the AWS Node SDK, we just need to set the options object that is passed as an argument to the upload method as follows:

const options: AWS.S3.ManagedUpload.ManagedUploadOptions = {
    partSize: 10 * 1024 * 1024, // each part is 10 MB
    queueSize: 2, // 2 parts are uploaded concurrently
};

const params: AWS.S3.PutObjectRequest = {
    Bucket: 'teste-multipart-upload',
    Key: key,
    Body: body,
};

return s3.upload(params, options, (err, data) => {
    if (err) throw err;
    console.log(data);
});

How can I do these two things at the same time? Either upload via SDK to a presigned url or upload a large file via the provided REST API (if there's another option I'd like to know).

Upvotes: 1

Views: 3345

Answers (2)

Albin
Albin

Reputation: 2558

You can also set it up so you handle it all in the browser safely with https://docs.aws.amazon.com/cognito/latest/developerguide/identity-pools.html

And then you can use S3.ManagedUpload(), which does all the work for you, instead of manually creating urls on the server.

(Note, this is TypeScript, but you can just remove " as HTMLInputElement" and change the import and if you want JS)

import AWS from "aws-sdk";

AWS.config.update({
  credentials: new AWS.CognitoIdentityCredentials({IdentityPoolId: "us-east-1:..."}),
  region: "us-east-1"
});
const s3 = new AWS.S3()

document.getElementById("file-input").addEventListener("change", event => {
  const inputElement = event.target as HTMLInputElement;
  const file = inputElement.files[0];
  const params = {
    Bucket: "BUCKET_NAME",
    Key: file.name,
    Body: file
  }

  const upload = new AWS.S3.ManagedUpload({params});
  upload.send((err, data) => {
    console.log({err, data});
  });
}, false);

Upvotes: 0

jarmod
jarmod

Reputation: 78573

This can be accomplished with a series of steps:

  1. The back-end initiates the multipart upload.
  2. The back-end creates a series of pre-signed URLs, one per part, and sends them to the client.
  3. The client uploads the various parts to those pre-signed part URLs (in any order).
  4. The client tells the back-end that it's done, and the back-end completes the multipart upload.

See Multipart uploads with S3 pre-signed URLs for an example.

Upvotes: 3

Related Questions