CODEWITHSUNDEEP
pythondjangoherokucelerydjango-celery
Jeff
Jeff

Reputation: 4037

Django Celery ConnectionError: Too many heartbeats missed

Question

How can I solve the ConnectionError: Too many heartbeats missed from Celery?

Example Error

[2013-02-11 15:15:38,513: ERROR/MainProcess] Error in timer: ConnectionError('Too many heartbeats missed', None, None, None, '')
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 97, in apply_entry
    entry()
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 51, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 153, in _reschedules
    return fun(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 265, in heartbeat_check
    return self.transport.heartbeat_check(self.connection, rate=rate)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 134, in heartbeat_check
    return connection.heartbeat_tick(rate=rate)
  File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 837, in heartbeat_tick
    raise ConnectionError('Too many heartbeats missed')
ConnectionError: Too many heartbeats missed

App Overview

Package Versions

Just what I think are relevant:

Django==1.4.3
amqp==1.0.8
billiard==2.7.3.20
celery==3.0.14
gevent==0.13.8
greenlet==0.4.0
kombu==2.5.6
raven==3.1.10

What I've Tried So Far

Possible Relevant Info

Upvotes: 11

Views: 7646

Answers (1)

asksol
asksol

Reputation: 19499

How often does it happen?

It may be that the heartbeat monitoring is not working properly in your case. The heartbeat support was introduced fairly recently, so there may be bugs. I cannot reproduce this here, so I need more data to understand what is going on.

You can disable heartbeats by setting BROKER_HEARTBEAT=0. If this is a bug then the worker should run fine, but it will not be able to quickly detect a broken connection. Being unable to detect connection loss is only a problem in some environments (usually caused by specific router/firewall configurations)

Upvotes: 9

Related Questions