CODEWITHSUNDEEP
node.jsamazon-web-servicesherokuamazon-s3
Murlidhar Fichadia
Murlidhar Fichadia

Reputation: 2609

Download pdf files from external url's - Heroku, NodeJS, Angular 7

I am trying to download multiple pdf files from external sources to my nodejs server (in Heroku) temporarily and upload it to AWS S3 bucket.

I have tried multiple methods all of which works fine in my local machine but not in Heroku Dyno NodeJS Server. I am unable to even create folder in Heroku. I guess due to limited permission.

In Node

1) using var download = require('download-file') (using this currently in below code)

2) axios

3) res.download()

Download Files Code

const downloadFiles = async (unique_files) =>  {

  for (let index = 0; index < unique_files.length; index++) {
    let file_ext = unique_files[index].substr(unique_files[index].length - 4);
      if(file_ext == ".pdf") {
        await downloadzz(unique_files[index])
      }
  }

}

function downloadzz(link) {
  download(link, function(err){ 
    if (err) throw err
    console.log("DOWNLOAD Complete");

  });
}

Upload Files Code

const uploadFiles = async (unique_files) =>  {

  for (let index = 0; index < unique_files.length; index++) {
    let file_ext = unique_files[index].substr(unique_files[index].length - 4);
      if(file_ext == ".pdf") {
        await uploadzz(unique_files[index])
      }
  }
}

function uploadzz(link) {
    fs.readFile(require('path').resolve(__dirname+'/../external-pdfs/', link.slice(link.lastIndexOf('/') + 1)), function (err, data) {

    params = {Bucket: pdfBucket, Key: link.slice(link.lastIndexOf('/') + 1), Body: data, ACL: "public-read" };
    s3.putObject(params, function(err, data) {
        if (err) {
            console.log("Failed Upload", err);
        } else {
            console.log("Successfully uploaded data to bucket", data);
        }
    });

  });
}

I don't get any error but no folder seem to exist with a name external-pdfs on heroku server.

I am open for better solutions: for example, directly uploading file from external url to s3... How can I in read file from a external url and directly upload to AWS S3 bucket?

Upvotes: 1

Views: 897

Answers (2)

vala keyur
vala keyur

Reputation: 19

In Angular, we can use FileSaver library to save the pdf file from library.

Find the below sample code to do this way. enter image description here

Upvotes: 0

alberto vielma
alberto vielma

Reputation: 2342

You can use axios. Setting the responseType as stream, you can get the file data and pass it as the body. Here it's an example code to get the pdf from a URL and uploading its info directly to S3:

const AWS = require('aws-sdk');
const axios = require('axios');

AWS.config.loadFromPath('./config.json');
const s3 = new AWS.S3({apiVersion: '2006-03-01'});
const URL = "<YOUR_URL>";

const uploadPdfToS3 = async () => {
    try{
        const {data, headers} = await axios.get(URL, {responseType: 'stream'});
        // Create params for putObject call
        const objectParams = {
            Bucket: "<YOUR_BUCKET>", 
            Key: "<YOUR_KEY>", 
            ContentLength: headers['content-length'],
            Body: data
        };
        // Create object upload promise
        await s3.putObject(objectParams).promise();
    } catch(err){
        console.log("ERROR --->" + err)
    }
}

Upvotes: 2

Related Questions