CODEWITHSUNDEEP
amazon-ec2message
ensnare
ensnare

Reputation: 42093

What are some good ways to handle Amazon EC2 queuing for thumbnail generation?

I am building a web-based photo gallery. Using Amazon EC2, I would like to be able to "spin up" machines for thumbnail generation when load is high, and spin them down when load is low.

What I am thinking is to have a database table of images that need to have their thumbnails generated, with fields like the below:

image_id (32-char string)
thumbs_generated (true or false)
currently_generating (true or false)
s3_key (32-char string)

When a new EC2 instance spins up, it'll connect to the database and retrieve an image where both thumbs_generated and currently_generating are set to false. Once the image is retrieved, currently_generating gets set to true. When the generation is complete, thumbs_generated is set to true and the script repeats with a new image.

Is this a good strategy? Any improvements or things to keep in mind from experience?

Would this be a good use case for Amazon's simple messaging service, or is that not necessary?

Thanks in advance.

Upvotes: 1

Views: 114

Answers (1)

Eran Sandler
Eran Sandler

Reputation: 491

You might be able to do so by using Amazon SQS (Simple Queue Service).

You will have to decouple your thumbnail generation into 2 pieces, one that sends a message to the SQS queue about which image to generate a thumb (as well as where that image exists) and a worker process running on different machines that polls the queue, generates the thumbs and report the success to whatever mechanism you require to report the result to (can be an additional queue or some other sync mechanism).

That way, when you have your worker processes on a different machine you can set an auto-scaling rule based on the size of the queue.

Check out this post, about someone who did a similar thing.

Upvotes: 2

Related Questions