CODEWITHSUNDEEP
pythondjangocelery
Omroth
Omroth

Reputation: 1129

Long celery task causes MySQL timeout in Django - options?

I have a celery task which takes about 6 hours. At the end of it, Django (or possibly Celery) raises an exception "MySQL server has gone away".

After doing some reading, it appears that this is a known issue with long tasks. I don't (think I have) control over pinging or otherwise mid-task; but the exception is raised after the call which takes time has finished (but still within the task function).

Is there a call I can make within the function to re-establish the connection?

(I have run this task "locally" with the same RDS MySQL DB and not had the issue, but I am getting it when running on an AWS instance.)

Upvotes: 1

Views: 1195

Answers (1)

Omroth
Omroth

Reputation: 1129

Eventually found what appears to have worked:

from django.db import close_old_connections
import time

def check_and_retry_django_db_connection():
    close_old_connections()

    db_conn = False
    while not db_conn:
        try:
            connection.ensure_connection()
            db_conn = True
        except OperationalError:
            print('Database unavailable, waiting 1 second...')
            time.sleep(1)

    print('Database available')

The key is the close_old_connections call - ensure_connection will not work otherwise.

Ian

Upvotes: 1

Related Questions