Psycopg2 Error When Running Django Via Serverless

Question

File "/var/task/django/db/backends/postgresql/base.py", line 29, in raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e) django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: libpq.so.5: cannot open shared object file: No such file or directory

I'm receiving the following error after deploying a Django application via Serverless. If I deploy the application via our Bitbucket Pipeline.

Here's the Pipeline:

    - step: &Deploy-serverless
        caches:
          - node
        image: node:11.13.0-alpine
        name: Deploy Serverless
        script:
          # Initial Setup
          - apk add curl postgresql-dev gcc python3-dev musl-dev linux-headers libc-dev

          # Load our environment.
          ...

          - apk add python3
          - npm install -g serverless

          # Set Pipeline Variables
          ...

          # Configure Serverless
          - cp requirements.txt src/requirements.txt
          - printenv > src/.env
          - serverless config credentials --provider aws --key ${AWS_KEY} --secret ${AWS_SECRET}
          - cd src
          - sls plugin install -n serverless-python-requirements
          - sls plugin install -n serverless-wsgi
          - sls plugin install -n serverless-dotenv-plugin

Here's the Serverless File:

service: serverless-django

plugins:
  - serverless-python-requirements
  - serverless-wsgi
  - serverless-dotenv-plugin

custom:
  wsgi:
    app: arc.wsgi.application
    packRequirements: false
  pythonRequirements:
    dockerFile: ./serverless-dockerfile
    dockerizePip: non-linux
    pythonBin: python3
    useDownloadCache: false
    useStaticCache: false

provider:
  name: aws
  runtime: python3.6
  stage: dev
  region: us-east-1
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - s3:GetObject
        - s3:PutObject
      Resource: "arn:aws:s3:::*"

functions:
  app:
    handler: wsgi.handler
    events:
      - http: ANY /
      - http: "ANY {proxy+}"
    timeout: 60

Here's the Dockerfile:

FROM lambci/lambda:build-python3.7
RUN yum install -y postgresql-devel python-psycopg2 postgresql-libs

And here's the requirements:

amqp==2.6.1
asgiref==3.3.1
attrs==20.3.0
beautifulsoup4==4.9.3
billiard==3.6.3.0
boto3==1.17.29
botocore==1.20.29
celery==4.4.7
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
coverage==5.5
Django==3.1.7
django-cachalot==2.3.3
django-celery-beat==2.2.0
django-celery-results==2.0.1
django-filter==2.4.0
django-google-analytics-app==5.0.2
django-redis==4.12.1
django-timezone-field==4.1.1
djangorestframework==3.12.2
Djaq==0.2.0
drf-spectacular==0.14.0
future==0.18.2
idna==2.10
inflection==0.5.1
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
kombu==4.6.11
livereload==2.6.3
lunr==0.5.8
Markdown==3.3.4
MarkupSafe==1.1.1
mkdocs==1.1.2
nltk==3.5
psycopg2-binary==2.8.6
pyrsistent==0.17.3
python-crontab==2.5.1
python-dateutil==2.8.1
python-dotenv==0.15.0
pytz==2021.1
PyYAML==5.4.1
redis==3.5.3
regex==2020.11.13
requests==2.25.1
sentry-sdk==1.0.0
six==1.15.0
soupsieve==2.2
sqlparse==0.4.1
structlog==21.1.0
tornado==6.1
tqdm==4.59.0
uritemplate==3.0.1
urllib3==1.26.3
uWSGI==2.0.19.1
vine==1.3.0
Werkzeug==1.0.1

And here's the database settings:

# Database Defintions
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "HOST": load_env("PSQL_HOST", "127.0.0.1"),
        "NAME": load_env("PSQL_DATABASE", ""),
        "PASSWORD": load_env("PSQL_PASSWORD", ""),
        "USER": load_env("PSQL_USERNAME", ""),
        "PORT": load_env("PSQL_PORT", "5432"),
        "TEST": {
            "NAME": "arc_unittest",
        },
    },
}

Am at a loss for what exactly the issue is. Thoughts?

---

File "/var/task/django/db/backends/postgresql/base.py", line 29, in raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e) django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2._psycopg'

I receive this similar error when deploying locally.

Answer 1

You can also use a lambda layer and use one already created in this repo

At the bottom of you Lambda console, you have a "Layers" section where you can click "Add Layer" and specify the corresponding ARN based on your Python version and AWS region.

It will also have the benefit of reducing your package size.

Answer 2

In my case, I needed to replace the psycopg2-binary with aws-psycopg2 to be Lambda friendly.

Answer 3

Apparently it is a dependency error, in debian base distributions you solve it by installing the libpq-dev package