CODEWITHSUNDEEP
javascriptnode.jsexpressamazon-s3
Catfish
Catfish

Reputation: 19284

Stream file upload to s3 via express server

I have an express endpoint where i currently handle uploading of files. Large files are taking lots of memory b/c i was using bodyParser which buffers the entire file in memory before calling my handler function.

I removed the bodyParser middleware from this endpoint and i'm strugging to properly use streams to basically stream the file upload -> express -> s3.

This is the docs on the s3 method and it accepts a buffer or a stream.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property

route

router.put('/files/:filename', putHandler({ s3Client: s3Client }))

I tried this which streams the file to my handler method, but it doesn't seem to be streaming it to the s3.upload method (no surprise really)

function put ({ s3Client }) {
  return (req, res) => {
        ...
        let whenFileUploaded = new Promise((resolve, reject) => {
          // const { Readable } = require('stream')
          // const inStream = new Readable({
            // read() {}
          // })

          let data = ''
          req.on('data', function (chunk) {
            req.log.debug('in chunk')
            data += chunk
            // inStream.push(chunk)
          })
          req.on('end', function () {
            req.log.debug('in end')
          })

          s3Client.upload(
            {
              Key: filepath,
              Body: data,
              SSECustomerAlgorithm: 'AES256',
              SSECustomerKey: sseKey.id.split('-').join('')
            },
            {
              partSize: 16 * 1024 * 1024, // 16mb
              queuSize: 1
            },
            (err, data) => err ? reject(err) : resolve(data)
          )
        })

My guess is that I need to create a stream and pipe req.on('data... to my stream and then set Body: inStream which you can see i attempted with the commented out stuff, but that didn't seem to work either.

Help?

Upvotes: 0

Views: 1848

Answers (1)

Catfish
Catfish

Reputation: 19284

Turns out the answer is actually very simple. All I had to do was pass the req object.

function put ({ s3Client }) {
  return (req, res) => {
        ...
        let whenFileUploaded = new Promise((resolve, reject) => {
          s3Client.upload(
            {
              Key: filepath,
              Body: req, // <-- NOTE THIS LINE
              SSECustomerAlgorithm: 'AES256',
              SSECustomerKey: sseKey.id.split('-').join('')
            },
            {
              partSize: 16 * 1024 * 1024, // 16mb
              queuSize: 1
            },
            (err, data) => err ? reject(err) : resolve(data)
          )
        })

The way i found this out is b/c I looked at the express source code for what a req object is and I see that it is a http.IncomingMessage object - https://github.com/expressjs/express/blob/master/lib/request.js#L31

Then i looked at the Node docs and I see that http.IncomingMessage implements the Readable Stream interface

It implements the Readable Stream interface, as well as the following additional events, methods, and properties.

https://nodejs.org/docs/latest-v9.x/api/http.html#http_class_http_incomingmessage

Upvotes: 2

Related Questions