CODEWITHSUNDEEP
amazon-s3aws-lambda
umdev
umdev

Reputation: 359

Lambda not invoking if the uploaded files are large in size in s3 bucket?

I have created a lambda which would invoke and do the transformation based on the event in the target source bucket.

This is working fine when I upload the small size of file in the targeted source bucket.

But when I upload large file(eg: 65 mb file), it looks lambda not invoking based on that event..

Appreciate if anyone can help on this kind of issue?

Thanks

Upvotes: 1

Views: 2900

Answers (3)

zabidima
zabidima

Reputation: 41

There could be any of these issues:

  1. Your event only captures s3:ObjectCreated:Put event, as others have mentioned. Usually if it's a big file, the event is s3:ObjectCreated:CompleteMultipartUpload instead. You could either:
    • add s3:ObjectCreated:CompleteMultipartUpload event to your capture, or
    • simply use s3:ObjectCreated:* event - this will include Put, MultiPart Upload, Post, Copy, and also other similar events to be added in the future (source: https://aws.amazon.com/blogs/aws/s3-event-notification/)
  2. Your Lambda function might run longer limit you set (limit is 15min).
  3. Your Lambda function requires more memory than the limit you set.
  4. Your Lambda function requires more disk space than the limit you set. This may be an issue if your function downloads the data on disk first and perform transformation there (limit currently is 512MB).

Upvotes: 2

Avishek Bhattacharya
Avishek Bhattacharya

Reputation: 6994

The large files in S3 are uploaded via S3 Multipart Upload instead of a regular PUT or single part upload process.

There can be two problems

  1. ``In your lambda you probably have created the subscription for s3:ObjectCreated:Put events. You should add s3:ObjectCreated:CompleteMultipartUpload too in the Lambda subscription list.

  2. Your lambda timeout could be small for and that works for the smaller files. You might want to increase that.

Upvotes: 2

Mayank Porwal
Mayank Porwal

Reputation: 34086

I am guessing, big files would be uploaded on S3 via S3 Multipart Upload instead of a regular put-object operation.

Maybe your Lambda function is just subscribed to s3:ObjectCreated:Put events. You need to add s3:ObjectCreated:CompleteMultipartUpload permission to Lambda as well.

Upvotes: 2

Related Questions