Deciphering a special case of psycopg2:OperationalError

Question

I have a Django web app with a postgresql backend where users congregate and message each other. These messages can be up or downvoted. If there's any abuse, people with a lot of downvotes get "ghost banned" (called hell banning on my website).

To accomplish this, I have a simple middleware that checks if a user is hell banned:

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
from myapp.models import HellBanList

class HellBannedMiddleware(object):

    def process_request(self, request):
        if request.user.is_authenticated():
            request.user_banned = HellBanList.objects.filter(condemned=request.user).exists()
        else:
            request.user_banned = 0

According to what the middleware returns, I take certain actions on the website.

I use newrelic to monitor my app. Intermittently, I see an error crop up. I need help in deciphering this and what steps I can take to handle it. It's as follows:

psycopg2:OperationalError

/myproject.middleware.HellBanned:HellBannedMiddleware.process_request

Error message

psycopg2:OperationalError: could not translate host name "mywebapp.cloudapp.net" to address: Name or service not known 

The full stack trace is as follows:

Stack trace
Traceback (most recent call last):
File "/home/myuser/.virtualenvs/myenv/bin/gunicorn", line 11, in <module>
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 189, in run
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 72, in run
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 181, in run
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 477, in manage_workers
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 540, in spawn_workers
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 124, in init_process
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 119, in run
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 66, in run_for_one
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 30, in accept
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/newrelic-2.56.0.42/newrelic/api/web_transaction.py", line 704, in __iter__
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/newrelic-2.56.0.42/newrelic/api/web_transaction.py", line 1080, in __call__
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 255, in __call__
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 92, in get_response
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/newrelic-2.56.0.42/newrelic/hooks/framework_django.py", line 228, in wrapper
File "/home/myuser/myfolder/myproject/middleware/HellBanned.py", line 8, in process_request
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/utils/functional.py", line 202, in inner
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/utils/functional.py", line 268, in _setup
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 18, in <lambda>
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 10, in get_user
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 134, in get_user
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 46, in __getitem__
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 168, in _get_session
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/user_sessions/backends/db.py", line 32, in load
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/models/manager.py", line 143, in get
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/models/query.py", line 382, in get
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/models/query.py", line 90, in __len__
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/models/query.py", line 301, in iterator
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 839, in execute_sql
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 326, in cursor
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 182, in _cursor
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/newrelic-2.56.0.42/newrelic/hooks/database_dbapi2.py", line 102, in __call__
File "/home/myuser/.virtualenvs/myenv/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect

If you see closely, there's an error on line 8 of my hell banning middleware, which happens to be this: if request.user.is_authenticated():

This error crops up intermittently like I said. Last time I saw a string of such errors was a few hours in the aftermath of a disk full error that caused my website to crash, although I've seen it pop up otherwise too.

Can anyone shed some expert light on what this is, and how I can handle the error for the unfortunate user who runs into this? Thanks in advance.

Answer 1

Background

It seems that you have a postgresql database hosted on mywebapp.cloudapp.net with it's port (5432) open to the public. This is not the best practice. Generally you should expose the database only on private network IPs for security reasons. Secondly connecting to a database on a remote location means you are going to add anything from a 20-30 milliseconds to a hundreds milliseconds to each query. If you fetch a large amount of data, the delay maybe measurable in seconds.

To make matters worse, you are using the FQDN of the server instead of it's IP address. That means each request also involves a DNS resolution. Unless DNS requests are cached locally that could be another 20-30 milliseconds overhead. And you also run into the problems that you just did!

Temporary solution:

To avoid your site failing due to DNS resolution problems, find the server's IP address and use it in your django settings.py rather than the host name.