CODEWITHSUNDEEP
node.jsamazon-s3aws-lambda
user1411254
user1411254

Reputation: 139

NodeJS csv data dump into s3 object

I'm a beginner with AWS, JavaScript and I'm trying to get data from an oracle database using the following code in a AWS Lambda function.

 let conn = await this.getConnection(userName, password, connectString);
 let results = await conn.execute(query, {}, {});
 let data = results.rows.map(el => `${el}`);
 console.log(data);

which gives me the following output on the console

[ "TEST_SESSION_0,121,High,Warning,7",
"TEST_SESSION_2,122,High,Error,11",
"TEST_SESSION_3,123,Low,Error,431,
"TEST_SESSION_4, ASMT_,103,Low,Fatal,10" ]

Now, my goal is to convert this output into CSV data stream and then upload into S3 bucket object output.csv for further processing. i tried below to convert into CSV but having issue to connect it back to S3 using AWS SDK and required NPM modules, when i test the below code i dont get any error but at the same time dont see data getting uploaded to S3. not sure where i'm doing wrong, appreciate any help..

stringify(results.rows, function(err, output){
  if(err){
    console.log('some error occured '+ err);
  } else {
     console.log(output); //gives me below output
//TEST_SESSION_0,121,High,Warning,7
//TEST_SESSION_2,122,High,Error,11
//TEST_SESSION_3,123,Low,Error,431
//TEST_SESSION_4, ASMT_,103,Low,Fatal,10

below code is to take CSV output to upload to S3 object

     var param = {
       Bucket: 'Test-Bucket',
       Key: 'output.csv',
      Body: new Buffer(fs.createWriteStream(output)) //can i pass the csv data that i captured above in "output"?
      };  
    s3bucket.putObject(param, function(err, output){
   if(err) console.log(err);
  else console.log(output);
});
}
})

Upvotes: 2

Views: 5674

Answers (1)

madebydavid
madebydavid

Reputation: 6517

I would start by separating out the code for the S3 upload and working on that part by itself - trying to get it to create a file in the bucket containing the text from a string.

The putObject function expects a Buffer as the Body param. To construct this from a string you can use the Buffer.from method:

var AWS = require('aws-sdk');

var output = 'This is a test';
var myBody = Buffer.from(output);

var param = {
  Bucket: 'Test-Bucket',
  Key: 'output.csv',
  Body: myBody
};

var s3bucket = new AWS.S3({ signatureVersion: 'v2' });
s3bucket.putObject(param, function(err, output) {
  if(err) {
    console.error(err);
  } else {
    console.log(output)
  }
});

Once you have that part working perfectly, plug it into the other code and change output to be the contents of the CSV file.

NB you will need the following environment variables set:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

Upvotes: 3

Related Questions