CODEWITHSUNDEEP
ruby-on-railsamazon-s3
CodeOverload
CodeOverload

Reputation: 48535

Best approach to poll for new S3 objects?

So i use S3 as an image store, where images are constantly uploaded to a bucket with various software.

I'm also building a rails app that lets people browse galleries of images on S3, but for performance issues I need to keep a copy of info about the images in the local database, so I don't have to connect to S3's API whenever someones views a gallery.

Now what is the best approach to get notified of new objects on S3 that i don't have info of on my db?

I thought making a worker that constantly fetches a list of all S3 objects and register the ones we don't have, but S3 has a limit of 1000 objects per API request and i couldn't find a way on the API docs to specify an offset for the next request until all objects are fetched.

Any one has an idea?

P.S. I'm using this gem to connect to S3

Upvotes: 2

Views: 2355

Answers (1)

j0nes
j0nes

Reputation: 8109

As stated in this answer here and in the S3 documentation, S3 does not allow notifications after uploading a new file to S3. With polling you will reach some limits sooner or later.

I would try a different approach. Go to your uploading software and add notifications there. For example you can send a notification to SNS that a new upload is being done. You can subscribe to this SNS queue then with a background task on your servers and fill the necessary infos in your database.

However if you want to go the polling way, have a look at the Marker attribute in your results object in the documentation page you linked. Every request returns a marker, and if you give this marker to the next request you will get the next objects that were not available in your first query.

Upvotes: 3

Related Questions