CODEWITHSUNDEEP
amazon-s3aws-lambdaamazon-sns
Kyle Heuton
Kyle Heuton

Reputation: 9768

How to get a single s3 file creation event

I have several amazon s3 buckets that different clients upload files to, typically using sftp. Because different clients access the buckets in different ways, I can't exactly anticipate how they will upload the files. I want to be notified when a new file arrives, so I have the buckets write to an SNS topic when any s3 object creation even is triggered. However, for some clients, this results in 3 different events for each file creation:

  1. An ObjectCreated:Put event with filesize 0
  2. An ObjectCreated:Copy event with the full filesize
  3. An ObjectCreated:CompleteMultiPartUpload, also with the full filesize.

I have a lambda function listening to this event, and I want it to only act once per file. Is there any to distinguish these duplicate events?

I don't want to have the s3 bucket only write about Copy or CompleteMultiPartUploads, because I worry that some clients will only trigger one of these (unless there is some guarantee that one of these triggers for every single s3 file creation, but I haven't seen that in the docs anywhere).

I don't really want to log received files in a database with my lambda, because that would make it challenging to detect when a client legitimately re-uploads a file with the same name.

I'm sure my lambda is working properly and these are distinct events, not retries

Upvotes: 5

Views: 2495

Answers (1)

Sid Moparthi
Sid Moparthi

Reputation: 33

You should only get one notification for a file. But in any case, for s3 file event triggers, you can use the option All object create events option. This will send one event per file.

Upvotes: 1

Related Questions