CODEWITHSUNDEEP
celery
User
User

Reputation: 2148

Multiple celery with redis

I'm running multiple celery instances (stage, dev, production) on a single Redis instance. Now the question is, does celery use pub/sub to communicate with redis? If no, does the isolation level can be achieved by using different db number? Eg: 

dev - redis://127.0.0.1:3366/0
stage - redis://127.0.0.1:3366/1
production - redis://127.0.0.1:3366/2

Thanks!

Upvotes: 2

Views: 1183

Answers (1)

Louis
Louis

Reputation: 151401

Quite irrespective of what Celery does with Redis, I would never have multiple Celery instances use the same Redis database. If multiple Celery instances are able to use the same Redis database harmoniously, that's great. But there is no telling whether a future version will have a bug that will cause havoc. Moreover, if you need to inspect what is going on on the Redis side, it is very convenient to be able to select a database number and know that anything you'll see in this database belongs to a single instance.

In fact, I've moved from using multiple databases to using multiple Redis instances. There has been talk in the past among the Redis developers of removing support for multiple databases. I believe this idea has been shelved but when the idea was being discussed the Redis developers made the point that running multiple Redis instances is really not expensive. Among the advantages of using multiple instances, the decisive factor for me was that a Redis instance started for development does not have the same configuration and management needs as a Redis instance started for production. For instance, it is okay to casually delete the data I have in my development instance. Not so for production. (Yes, it is a cache and the production data would be repopulated but I don't want to flush the whole cache unless there's no other choice.) If I decide I want to zap all the Redis data associated with development, including whatever has been saved to disk for persistence, I can trivially do it and know that it is not affecting production. I've had to do it a couple of times and was immensely glad that having a separate Redis instance for development made the operation a no-brainer.

Upvotes: 4

Related Questions