OperationalError: cursor "_django_curs_<id>" does not exist

Question

We have an online store web-app which is powered by django, postgresql and heroku.

For a specific campaign (you can think a campaign like a product to purchase), we have sold 10k+ copies successfully. Yet some of our users are encountered this error according to our Sentry reports. Common specification of these users is; none of them have address information before the purchase. Generally, users fill out address form right after registering. If they don't, they need to fill the form while purchasing the product and submit them together.

This is how the trace looks like:

OperationalError: cursor "_django_curs_140398688327424_146" does not exist

(66 additional frame(s) were not displayed)
...
  File "store/apps/store_main/templatetags/store_form_filters.py", line 31, in render_form
    return render_to_string('widgets/store_form_renderer.html', ctx)
  File "store/apps/store_main/templatetags/store_form_filters.py", line 20, in render_widget
    return render_to_string('widgets/store_widget_renderer.html', ctx)
  File "store/apps/store_main/widgets.py", line 40, in render
    attrs=attrs) + "<span class='js-select-support select-arrow'></span><div class='js-select-support select-arrow-space'><b></b></div>"

OperationalError: cursor "_django_curs_140398688327424_146" does not exist 

So another weird common thing, there are exception messages between sql queries before the failure. You can see it in the image below:

I'm adding it if they are somehow related. What may also be related is, the users who get this error are the users who tries to purchase the campaign right after a bulk mailing. So, extensive traffic might be the reason yet we are also not sure.

We asked Heroku about the problem since they are hosting the postgres, yet they do not have any clue either.

I know the formal reason of this error is trying to reach the cursor after a commit. Since it is destroyed after transaction, trying to reach it cause this error yet I don't see this in our scenario. We are not touching the cursor in any way. What am I missing? What may produce this error? How to prevent it? Any ideas would be appreciated.

Answer 1

I found that other postgres errors can be the cause of this cursor error, just like Rômulo said above.

For me, I was simply missing a table which caused an error, which canceled the transaction, which causes the server-side cursor to be canceled (see postgres docs). This error scrolled out of view when pytest code further down the stack complained about the missing cursor.

TLDR: Scroll up before you hit StackOverflow :-)

Answer 2

I had the same problem and realized that it had nothing to do with migrations by analyzing the traceback

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
    cursor.execute(sql, params)
  File "/opt/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/opt/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: temporary file size exceeds temp_file_limit (1537530kB)
ERROR:  temporary file size exceeds temp_file_limit (1537530kB)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.ConfigurationLimitExceeded:  (1537530kB)

The cause of the failure was Postgres configuration temp_file_limit, which was making it not possible to create the cursor because it

Limits the total size of all temporary files used by each process Specifies the maximum amount of disk space that a process can use for temporary files, such as sort and hash temporary files, or the storage file for a held cursor. A transaction attempting to exceed this limit will be canceled. If this value is specified without units, it is taken as kilobytes. -1 (the default) means no limit. Only superusers and users with the appropriate SET privilege can change this setting.

And the other error happens while trying to handle this exception.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/app/manage.py", line 20, in <module>
    main()
  File "/usr/src/app/manage.py", line 16, in main
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/django/core/management/commands/dumpdata.py", line 265, in handle
    serializers.serialize(
  File "/opt/venv/lib/python3.11/site-packages/django/core/serializers/__init__.py", line 134, in serialize
    s.serialize(queryset, **options)
  File "/opt/venv/lib/python3.11/site-packages/django/core/serializers/base.py", line 132, in serialize
    for count, obj in enumerate(queryset, start=1):
  File "/opt/venv/lib/python3.11/site-packages/django/core/management/commands/dumpdata.py", line 222, in get_objects
    yield from queryset.iterator()
  File "/opt/venv/lib/python3.11/site-packages/django/db/models/query.py", line 516, in _iterator
    yield from iterable
  File "/opt/venv/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1563, in execute_sql
    cursor.close()
psycopg2.errors.InvalidCursorName: cursor "_django_curs_..._sync_8" does not exist

Answer 3

Most likely you have forgotten to makemigrations and migrate them to the database. If you are sure you did make the migrations and running python manage.py makemigrations and python manage.py migrate will not find the changes you made then the database and models are not in sync. Sometimes this situation can be very frustrating if you have a big database. You will need to manually inspect your models.

To help out you can try this trick, which has been working for me.

• Step 1 (delete all migrations from apps) I am going to assume you are using the unix terminal. run sudo rm -rv */migrations/*. This removes all the migrations files and caches.
• Step 2 (make the migration folders in each app) run the command mkdir <app-folder>/migrations && touch <app-folder>/__init__.py. Replace with the name of the app that you have in the INSTALLED_APPS list in your default django settings file.
• Step 3 (Make Migrations) Here we populate the migrations folders in each app with migration files. Run python manage.py makemigrations. Run the second command. python manage.py migrate --fake. We are using the --fake flag because the data already exists in the database and so we do not want to populate name database tables that are already there or you will be greeted with the already exists error

If this did not work, then you will need to temper with some additional fields on the database. such as the django_migrations or a similarly named table. This is not recommended as there are other database tables that depend on it such as the django_contenttypes and this will throw you in a chain of related tables that you will manually inspect which is very painful.

So here is another trick you could try if you would want to mess with the database directly. So, for this, you will need to have correct permissions to edit the database as a user. If you are using sqlite3 as your database, you would want to use a tool like sqlite tool.

Okay, so once in the database that you are using for your Django project, look for a relation or table named django_migrations. Drop/Delete the table. In normal instances, it would fail to delete due to foreign key contraints, delete every database relation/table that is related to the django_migrations table. These tables include django_contenttypes, auth_permissions (and any other that I have missed). No need to worry becuase you are not tinkering with the important actual web application data. Once deleted, run migrations with python manage.py makemigrations and python manage.py migrate, in an active virtual environment.

This may most likely get rid of that error.

Answer 4

If you're using django-pytest and have enabled the optimization --reuse-db and have made DB migrations between test runs you need to re-create the DB tables again.

pytest --create-db

Answer 5

The reason for your error might be that you added fields to a model and forgot to makemigrations and migrate.

Have a look at this answer: When in run test cases then I will get this error: psycopg2.OperationalError: cursor "_django_curs_140351416325888_23" does not exist