CODEWITHSUNDEEP
pythonredisrabbitmqcelerymessagebroker
Vanjith
Vanjith

Reputation: 540

Is it feasible to often switching between redis and rabbitmq in celery?

In production we facing some unknown errors occuring in celery when we use redis as message broker. So we thought of migrate to rabbitmq until the errors get fixed. So that in future if there is any error in any one of them we can quickly switch between them. Is this idea feasible and is it possible to implement?

Thanks in advance

Upvotes: 0

Views: 871

Answers (1)

sytech
sytech

Reputation: 40941

In general, yes, transports are interchangeable, with some caveats. Celery will work the same way when you swap between supported brokers. It's important, however, to know what contracts Celery offers you when you use it and which behaviors may be broker-specific.

Caveats:

  1. Some celery settings/features depend on a specific transport. (for example, the broker_use_ssl setting is only valid with redis and amqp, not rabbitmq)

  2. Different brokers have different default settings/behavior (for example, redis defaults to 1 hour visibility timeout and SQS uses 30 seconds by default)

  3. In addition to differences with defaults, your broker could be configured to behave differently completely independently of your application configuration / celery settings (for example, queue settings when creating an SQS queue).

  4. You have to consider potential for data/message loss. Any in-flight messages or messages on the queue will be lost when you switch. So you'll want to make sure you handle the swap gracefully to avoid losing messages -- or in other words: you need to migrate or redeliver any existing messages when you switch between transports. Similarly, existing rate limit counters will not transition over, message deduplication mechanisms will not transfer, etc.

  5. Workers must be restarted for broker change to take effect. Settings cannot be changed 'on-the-fly' or automatically in response to errors.

So, yes, you can change transports whenever you want in general. However, it's probably not a good strategy for added fault tolerance or automated failover, particularly because of caveats (4) and (5).

Upvotes: 1

Related Questions