Where to run Celery on AWS

Question

In my django web app I am running a calculation which takes 2-10 minutes, and the AWS server times out with a 504 server error.

It seems that the best way to fix this problem is to implement a worker, and offload the calculations. Through some research, it seems that Celery with a Redis server (maybe AWS SQS instead?) are best for Django.

However, I only see tutorials for instances that are run locally. The Redis server is hosted by Railway and Celery is run in a separate terminal than Django (on the same local machine). I'm curious if/where Celery and Redis should be running on AWS.

This answer says you should try to run celery as a deamon in the background. AWS Elastic Beanstalk uses supervisord already to run some deamon processes. So you can leverage that to run celeryd and avoid creating a custom AMI for this. It works nicely for me. but don't the Celery and Redis servers still need to run somehow?

Where does the Celery server run? How do I start it? How can I leverage supervisord to run daemon processes? The documentation isn't helping me very much with AWS integration

Answer 1

You can configure Procfile to run multiple processes like main django app, celery and celery-beat in parallel as documented here:

web: <command to start your django app>

celery: celery -A <path_to_celery_app> worker

celery_beat: celery -A <path_to_celery_app> beat