David Sigley
David Sigley

Reputation: 1198

If my RPC logic with Django and Pika is inactive for too long, I lose the DB connection and subsequent requests fail with psycopg.OperationalError

The error I get is

psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly

I am using RabbitMQ as my message broker, though I don't think it's related.

Ultimately, I believe the problem is having a very long running thread doing nothing and the DB connection being terminated.

I have tried forcing reopening the DB connection with:

django.db.connection.connect()

and more recently, I've tried closing and opening connections before and after executing the RPC with:

def close_database(message: str) -> None:
  print(f"Attempting to close database connections: {message}")
  for connection in connections.all():
    try:
        connection.close()
        print(f"Closed connection: {message}")
    except InterfaceError:
        pass
    except DatabaseError as exc:
        str_exc = str(exc)
        if "closed" not in str_exc and "not connected" not in str_exc:
            print(f"Error closing connection: {exc}")

def handle_rpc_request(routing_key: str, data: dict[str, Any]) -> Optional[str]:
    """Handle an RPC request."""
    print(f"External message received in Pulse - {routing_key}")

    # Another package had a similar issue and fixed it this way. See this PR:
    # https://github.com/mozilla/telemetry-analysis-service/pull/246/files
    close_database("before RPC request")
    try:
        response = celery_pubsub.publish_now(routing_key, data)
    finally:
        close_database("after RPC request")

Neither of these solutions are working for me.

I'm using

I would appreciate help with either of these 2 points:

  1. Any idea what else I can try in order to resolve the DB connection being lost?
  2. If not, how can I recreate this issue so I don't have a multi-day feedback loop before knowing if my solution has worked?

Thanks in advance!

Upvotes: 2

Views: 46

Answers (0)

Related Questions