CODEWITHSUNDEEP
pythonmongodbdockerdocker-composeairflow
Wesley
Wesley

Reputation: 31

ModuleNotFoundError: No module named 'pymongo' with Docker and Airflow

I'm currently using Docker with puckel/Airflow to run Airflow I installed pymongo successfully but when calling the import of pymongo, it still fails to find the module.

I added below codes into the Dockerfile above the other RUN before rebuilding

1st attempt

RUN pip install pymongo

2nd attempt

RUN pip install pymongo -U

I built them with

docker build --rm -t puckel/docker-airflow .

Pymongo does install successfully but when I do run the webserver with a simple import of dags I still get the error

 File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/mongo_hook.py", line 22, in <module>
     from pymongo import MongoClient
 ModuleNotFoundError: No module named 'pymongo'

Upvotes: 3

Views: 4346

Answers (5)

F Blanchet
F Blanchet

Reputation: 1510

I solved it by copying my requirements.txt file in the root.

In fact in puckel/docker-airflow's Dockerfile it executes entrypoint.sh witch pip install packages from /requirements.txt if the file exists. So we are sure our packages are installed.

You can add in the Dockerfile :

COPY ./requirements.txt /requirements.txt

Or

in a docker-compose.yml add a volume to your container:

volumes:
  - ./requirements.txt:/requirements.txt

Upvotes: 2

Kevin Postlewaite
Kevin Postlewaite

Reputation: 615

I ran into this same symptom. I fixed it by adding && pip install pymongo \ to puckel/airflow:Dockerfile, near the other pip install commands and rebuilding the image.

Here's what I tried that did not fix the problem:

  1. Adding pymongo to requirements.txt and mounting the file. I verified that the module was loaded as expected via log messages in docker-compose startup and by connecting to my worker and webserver instances and seeing that the module was available in the Python environment using help("modules") but the module was not available to my Airflow DAGs
  2. Adding --build-arg PYTHON_DEPS="pymongo" as a parameter to my docker build command. (Note: for modules other than pymongo this step fixed module not found errors, but not for pymongo. In fact, I did not see any log record of pymongo being installed during docker build when I set this)

Upvotes: 1

Yong Wang
Yong Wang

Reputation: 1313

I has similar experience for mysql hook and solved.

My experience is check if the module could be imported in pure python enviroment first.

Some time, the pack you installed is not the airflow wanted.

For your case, you could check in following step. 1. jump into the docker container docker exec -it /bin/bash 2. launch python assuming you has use python 3.X version python 3. check the module in python enviromnet import pymonggo # other test script if you to check. if you facing error, pls solve it in python environment first and then go back to airflow.

======================================================= I Just double checked the airflow github source code and realized that mongo db is not default hook in original source code.

In case case, you might need go further into pymongo package to study how to install & compile it and related dependence.

Upvotes: 0

Tomme
Tomme

Reputation: 321

When you built the puckel/Airflow Docker image, did you add mongo to AIRFLOW_DEPS in your build arguments?

e.g. docker build --rm --build-arg AIRFLOW_DEPS="mongo" -t puckel/docker-airflow .

Upvotes: 0

Raj006
Raj006

Reputation: 562

Could you, try

RUN pip3 install pymongo

and report back. It might happen if you have multiple versions of Python. pip3 will make sure you are installing the module for Python 3.x.

Upvotes: 0

Related Questions