CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-sdkaws-sdk-nodejs
GCSDC
GCSDC

Reputation: 3500

AWS S3 Readstream returns no data

I have an application on which files are uploaded to S3, and an event is triggered to process them by a Lambda function.

When a file is uploaded I may see the function execution on Cloud Watch logs, however no data is returned, no error is thrown and the on('end') handler is never called.

The files being processed are .csv, and I'm able to open them and check the contents manually.

Any ideas on what may be happening?

This is my code:

let es = require('event-stream');
let readStream = s3.getObject({
  Bucket: event.Records[0].s3.bucket.name,
  Key: event.Records[0].s3.object.key
}).createReadStream();

readStream
.pipe(es.split())
.pipe(es.mapSync(function (line) {
  console.log(line);
  processLine(line);
}))
.on('end', async () => {
  console.log('ON END');
  callback(null, 'OK');
})
.on('error', (err) => {
  console.error(JSON.stringify(err));
  callback(JSON.stringify(err));
})

Upvotes: 0

Views: 1023

Answers (1)

Joey Kilpatrick
Joey Kilpatrick

Reputation: 1602

When a Node.js Lambda reaches the end of the main thread, it ends all other threads.

Make your handler async and then promisify the last call as

new Promise((resolve) => {
    readStream
    .pipe(es.split())
    .pipe(es.mapSync(function (line) {
      console.log(line);
      processLine(line);
    }))
    .on('end', async () => {
      console.log('ON END');
      callback(null, 'OK');
    })
    .on('error', (err) => {
      console.error(JSON.stringify(err));
      callback(JSON.stringify(err));
      resolve();
    })
}

Upvotes: 1

Related Questions