CODEWITHSUNDEEP
dockerairflowminio
VKarthik
VKarthik

Reputation: 1429

minio ModuleNotFoundError when running Airflow Docker image

I have downloaded the latest Docker image for the Airflow and am able to spin up the instance succesfully. On my local system I have installed minio server using homebrew on my Mac.

I have created a DAG file to upload data to my Minio bucket. I have done a sample upload using python and it is working as expected (using the minio python libraries). On the Airflow server I am seeing the following errors -

ModuleNotFoundError: No module named 'minio'

Can someone pleae help me how can I have the pip3 minio library to the docker container so that this error can be resolved? I am new to containers and would really appreciate a easy guide or link that I can refer to help me with this error.

One of the things I did try is to fiddle with the attribute - _PIP_ADDITIONAL_REQUIREMENTS that comes in the AIRFLOW DOCKER image following this link but to no avail.

I added the values as - minio but didn't work.

Upvotes: 0

Views: 838

Answers (1)

ozs
ozs

Reputation: 3681

you can create a Dockerfile that extend the basic airflow and install your packages.

  1. Create Dockerfile
FROM apache/airflow:2.3.0

USER root

RUN apt-get update

USER airflow

RUN pip install -U pip

RUN pip install --no-cache-dir minio # or you can copy requirments.txt and install from it
  1. Build your docker

docker build -t my_docker .

  1. Run the new docker image (if you are using the docker-compose then change the airflow image to your image)

Upvotes: 0

Related Questions