CODEWITHSUNDEEP
pythondjangoamazon-ec2rediscelery
pass-by-ref
pass-by-ref

Reputation: 1298

Redis - Celery Configuration over Amazon EC2

I am developing a web application using Python, Django and MySql. I have a provision in the application where in a user can upload *.wmv and *.mov files which the system will process and convert it into *.mp4. I was using a single server architecture but as the user base is growing, video conversion consumes 90% of the memory. I am thinking of moving the video conversion / streaming server over cloud using Amazon EC2 so that the conversion takes place on a different server using Redis and Celery. I wanted to know if Redis - Celery supports multi-tier architecture. If anyone has ever successfully achieved this, please let me know the steps. It will be really helpful as I was googling this but couldn't find any supporting documentation.

Upvotes: 0

Views: 3679

Answers (1)

Malcolm Box
Malcolm Box

Reputation: 4036

If you want to use a Celery queue to process your jobs, you'll need to:

  • Set up rabbitMQ. You can do that on your existing host or on an EC2 instance
  • Set up Celery. The Docs are good.
  • Set up redis for whatever you need it for. Again, the docs are good
  • Configure your existing server to be able to talk to the task queue and redis. Basically, tweak EC2 firewall rules
  • Make sure the worker server(s) can see the files. Upload them to S3 is probably the easiest way to shift them back/forth from the cloud
  • Change your webserver code to trigger a celery task to do the conversion
  • Figure out what to do when the conversion finishes - how do you let the user know it's done?
  • Go.

Don't worry about "n-tier" etc etc - this is a simple setup that you can get going in a day by following the docs.

Upvotes: 5

Related Questions