Django occassionally and temporarily cant connect to database. It is causing requests to it to temporarily timeout and fail

Question

Setup: AWS RDS db (postgres), Dockerised django backend hosted on elastic beanstalk

Some context:

I have a django backend that accepts form post requests from a separate website, and django saves the form data to the database. The django app I'm creating is replacing existing software that is used for this purpose. I started testing the django app by having the website send the form data to BOTH the existing software and my django app. I noticed that occassionally my django app will not receive a form that the existing software receives. E.g django will finish the day with 25 forms but the existing software will finish with 26. I have request logging setup, but even the logs don't show any records of the 26th form.

To debug what was going wrong, I set up a script command within the django app to send forms to my django app endpoint every minute. It eventually failed:

 Exception retrieving data fields


SSL SYSCALL error: EOF detected

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/src/leads/management/commands/bug_finder.py", line 115, in handle
    data_field = DataField.objects.get(api_field=field)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: SSL SYSCALL error: EOF detected

[DJANGO] DEBUG 2022-09-08 22:43:09,567 connectionpool urllib3.connectionpool._new_conn:1003: Starting new HTTPS connection (1): url:443
[DJANGO] DEBUG 2022-09-08 22:43:09,730 connectionpool urllib3.connectionpool._make_request:456: https://url:443 "POST /new_lead/5kbLWVNXHQ3U6lJ9trME8hPSf HTTP/1.1" 400 799

So it seems like django is unable to connect with the database to perform queries/saves sometimes.
My local script exception occurred when trying to access datafield models from the database to generate a fake payload for the form:

            payload = {}
            for field in api_fields:
                try:
                    data_field = DataField.objects.get(api_field=field)
                    payload[field] = generate_value(data_field)
                except DataField.DoesNotExist:
                    continue

So I guess sometimes when requests are made from the website, django cant connect with the db, it raises this exception, doesnt save the form, and fails to send a response to the website.

When searching the above exception, most answers seem to suggest to relate to persistent db connection setups that have improper timeout settings, which I don't think relates to django request response system.

Is this normal, is it due to AWS going down? Would a solution be to wrap the endpoint's view in a try catch for this exception and try to store the form data in cache until django can connect with the DB again? is there an easy way to test whether django is connected to the db?

Something like a wrapper that effectively does:

try:
    view_function()
except psycopg2.OperationalError:
    counter = 0
    while counter <= 3:
        if django.is_connected_to_db:
            return view_function()
        else:
            counter += 1
            time.sleep(20)
    save_request_to_cache_to_try_later
    return response

Seems like I shouldn't have to be doing this though

Answer 1

For your question regarding checking the database connection in Django, you can use this code wherever you want:

from django.db import connections
from django.db.utils import OperationalError
try:
    db_conn = connections['default']
except OperationalError:
    time.sleep(20)

About the database problem: Which region are you exactly using on AWS? I have deployed multiple projects on us-east-1 and eu-central-1 and didn't have the problem.

This problem seems unusual, and you should send a ticket to AWS support about that. Anyway, you can try migrating your RDS to the Elastic Beanstalk managed databas. It is basically the same as normal RDS, but the network configuration is easier. Just make sure that RDS is in the same VPC as your instances.