CODEWITHSUNDEEP
pythondjangopostgresqlpytestgevent
Michal Charemza
Michal Charemza

Reputation: 27022

StreamingHttpResponse: return database connection to pool / close it

If you have a StreamingHttpResponse returned from a Django view, when does it return any database connection to the pool? If by default it does it once the StreamingHttpResponse has completed, is there a way to make return the connection earlier?

def my_view(request):
  # Some database queries using the Django ORM
  # ...

  def yield_data():
    # A generator, with no database queries using the Django ORM
    # ...

  return StreamingHttpResponse(
    yield_data(), status=200
  )

If it makes a difference, this is using https://pypi.org/project/django-db-geventpool/ with gunicorn, and any answer should also work when tested with pytest.mark.django_db (that I think wraps tests in transactions)

Upvotes: 5

Views: 1082

Answers (1)

Tarun Lalwani
Tarun Lalwani

Reputation: 146620

If you look at the documentation

https://docs.djangoproject.com/en/3.0/ref/databases/

Connection management

Django opens a connection to the database when it first makes a database query. It keeps this connection open and reuses it in subsequent requests. Django closes the connection once it exceeds the maximum age defined by CONN_MAX_AGE or when it isn’t usable any longer.

In detail, Django automatically opens a connection to the database whenever it needs one and doesn’t have one already — either because this is the first connection, or because the previous connection was closed.

At the beginning of each request, Django closes the connection if it has reached its maximum age. If your database terminates idle connections after some time, you should set CONN_MAX_AGE to a lower value, so that Django doesn’t attempt to use a connection that has been terminated by the database server. (This problem may only affect very low traffic sites.)

At the end of each request, Django closes the connection if it has reached its maximum age or if it is in an unrecoverable error state. If any database errors have occurred while processing the requests, Django checks whether the connection still works, and closes it if it doesn’t. Thus, database errors affect at most one request; if the connection becomes unusable, the next request gets a fresh connection.

Also if you see the db/__init__.py in django source code

# For backwards compatibility. Prefer connections['default'] instead.
connection = DefaultConnectionProxy()


# Register an event to reset saved queries when a Django request is started.
def reset_queries(**kwargs):
    for conn in connections.all():
        conn.queries_log.clear()


signals.request_started.connect(reset_queries)


# Register an event to reset transaction state and close connections past
# their lifetime.
def close_old_connections(**kwargs):
    for conn in connections.all():
        conn.close_if_unusable_or_obsolete()


signals.request_started.connect(close_old_connections)
signals.request_finished.connect(close_old_connections)

It has connection to the request_started and request_finished signal to close old connections using close_old_connections.

So if you are not willing to wait for connections to be closed you can call this method yourself. You updated code will be like below

from django.db import close_old_connections

def my_view(request):
  # Some database queries using the Django ORM
  # ...
  close_old_connections()

  def yield_data():
    # A generator, with no database queries using the Django ORM
    # ...

  return StreamingHttpResponse(
    yield_data(), status=200
  )

Upvotes: 5

Related Questions