CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
RJV
RJV

Reputation: 41

summary of s3 objects created daily

I have a process which will load all files to an S3 bucket. I subscribed(SNS) to the bucket which triggers an email for each file. However, my process is loading over a 10000 files everyday.

What is the best way to summarize and just have one email notification(at a certain time of the day) with total files uploaded. Also, send an email when no files are received by that time?

Upvotes: 0

Views: 999

Answers (2)

David Parks
David Parks

Reputation: 32081

You shouldn't do a list bucket operation daily, that would be wasteful on a large bucket, use the S3 inventory to generate the daily inventory then either compare against the previous day or use the timestamps in the CSV.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

Upvotes: 1

John Rotenstein
John Rotenstein

Reputation: 269548

You should schedule a daily run of an AWS Lambda function that will:

  • List the contents of the S3 bucket
  • Count the number of objects with a LastModified within the previous 24 hours
  • Send a notification via Amazon SNS

If the bucket is receiving 10,000+ objects each day, it can take a considerable time to list the contents of the bucket. Make sure the Lambda function has a suitably high timeout since it can only list 1000 objects per API call.

Upvotes: 0

Related Questions