CODEWITHSUNDEEP
pythondata-structuresamazon-s3queuedistributed
TIMEX
TIMEX

Reputation: 272094

If users are uploading images, and I need to do a lot of resizing/uploading, how should I set up my queue?

When the user POSTS to my server, should I store the picture in the body of the queue, and then have the "worker" servers pull it out and resize/upload to S3?

The reason I'm using a queue is because resizing/uploading 20 images to S3 takes a long time.

Upvotes: 0

Views: 84

Answers (1)

Zack Bloom
Zack Bloom

Reputation: 8417

Amazon's Simple Queue Service may be used to maintain communication between your worker and frontend servers, but keep in mind that it only supports messages up to 64kb, meaning it won't be possible to keep the images in the queue. This is probably best, filesystems are designed to maintain large files, queue implementations generally aren't.

I would have the user upload directly to S3, then use the SQS to coordinate communication between your worker servers as they process the images and return them to S3.

Upvotes: 1

Related Questions