CODEWITHSUNDEEP
javascriptnode.jsamazon-s3
captainill
captainill

Reputation: 2089

How to stream file to s3 from url

How do I grab a file from a url and then upload to s3? There are alot of examples using Multipart Upload out there but this should be simpler.This code executes server side. S3 upload fails with 'NetworkingError: Cannot pipe. Not readable.' 

import http from 'http';
import fs from 'fs';
import AWS from 'aws-sdk';
AWS.config.region = 'us-west-1';

const file = fs.createWriteStream('file.jpg');
var request = http.get('http://weknowmemes.com/wp-content/uploads/2012/04/dont-worry-cat-it.jpg', function(response) {
  response.pipe(file);

  file.on('finish', function() {
    // Upload the File
    var s3 = new AWS.S3({params: {Bucket: Config.s3Bucket}});
    var params = {Key: 'mykey', ContentType: 'image/jpg', Body: file};
    s3.putObject(params, function (err, data) {
      console.log('err', err)
      console.log('data', data)
    });
  });
});

Upvotes: 2

Views: 2938

Answers (1)

captainill
captainill

Reputation: 2089

Alright I solved this. Here's my working solution:

import request from 'request';
import bl from 'bl';
import AWS from 'aws-sdk';

const regexp = /filename=\"(.*)\"/gi;
let fileName;
let mimeType;

request
  .get(imageUrl)
  .on('response', function(response) {
    fileName = regexp.exec(response.headers['content-disposition'] )[1];
    mimeType = response.headers['content-type'];
  })
  .pipe(bl(function(error, data) {
    const s3 = new AWS.S3({params: {Bucket: Config.s3Bucket}});
    const params = {Key: fileName, ContentType: mimeType, Body: data};
    s3.putObject(params, function(s3Error, data) {
      if (s3Error) {
        console.log('Error uploading data: ', s3Error);
      } else {
        console.log('Successfully uploaded data: ', data);
      }
    });
  }))

I didn't need to use a WriteStream instance but I did need to get the raw data binary for my image in order to assign to the Body parameter of s3's putObject method. This required using a BufferList ie. import bl from 'bl';

Really my main issue was not understanding how to get the data formatted the right way in from request.get and out to S3. When you are using a Multipart Form it handles the file for you hence all the simplish examples I kept finding for that use case.

Upvotes: 1

Related Questions