CODEWITHSUNDEEP
amazon-web-servicesredisceleryamazon-elasticache
user1187968
user1187968

Reputation: 7986

AWS + Celery + ElastiCache (Redis Cluster) Error: CROSSSLOT Keys in request don't hash to the same slot

I have a Dockerized Celery working on AWS, and it's using an ElastiCache (Redis Cluster with multi-nodes) from AWS as the message broker, but I'm getting the following error.

When I test my Celery workers on my local machine, it talked to a single node Redis totally fine. How should I fix this problem? 

[I 180518 18:54:20 mixins:224] Connected to redis://....use1.cache.amazonaws.com:6379//

[E 180518 18:54:20 events:123] Failed to capture events: 'CROSSSLOT Keys in request don't hash to the same slot', trying again in 1 seconds.

...

[E 180518 18:54:23 events:123] Failed to capture events: 'Command # 1 (LLEN celeryev....2d788) of pipeline caused error: MOVED 11904 10.0.x.xxx:6379', trying again in 1 seconds.

Upvotes: 2

Views: 3671

Answers (2)

fudu
fudu

Reputation: 742

So far, the Celery doesn't support for ElastiCache Redis in Cluster mode, so either disable it or change to some other supported message broker such as RabbitMQ or AWS SQS.
And because in the production environment, Cluster mode is highly recommended because it could keep the High Availability for the site, so it better to not turn it off but switch to other message brokers.

I've tried to make it work by customizing Celery worker based on this package, but not working:
https://github.com/hbasria/celery-redis-cluster-backend

Here's the repo that I've tried to apply the above package:
https://github.com/congson95dev/celery-tutorial/tree/celery-redis-cluster-mode

It end up with dead end with these errors:
ResponseError('CLUSTERDOWN The cluster is down') or redis.exceptions.ResponseError: MOVED 192.168.80.7:6379

=> So to conclude, please change to some other supported message broker such as RabbitMQ or AWS SQS
Here's the example of how to setup Celery using AWS SQS:
https://github.com/congson95dev/celery-tutorial/tree/celery-sqs

Upvotes: 0

Lucas Carvalho
Lucas Carvalho

Reputation: 101

This error occurs because keys must be in the same hash slot and not just the same node. To implement multi-key operations in a sharded Redis (cluster mode enabled) ElastiCache cluster, the keys must be hashed to the same hash slot. You can force keys into the same hash slot by using hash tags.

I also recommend to check if your application support Redis using Cluster mode

More info here and here

Upvotes: 0

Related Questions