CODEWITHSUNDEEP
pythonpython-3.xairflowrequirements.txt
Romsik788
Romsik788

Reputation: 65

How to install dependencies from requirements.txt in Apache Airflow

I have a kubernetes cluster with Apache Airflow deployed. I used the official helm chart to deploy it. The DAG is stored in the git repository with which Apache Airflow is synchronized (using the dags.gitSync option). Question: How can I get Apache Airflow to install the dependencies from the requirements.txt itself? This file is located at the following path:

dags
└── repo
    └── accounting
        ├── dag.py
        └── requirements
            └── requirements.txt

Apache Airflow version: 2.3.0

Upvotes: 3

Views: 5605

Answers (2)

S N
S N

Reputation: 600

You can extend Airflow image like Jongwho Park has mentioned, alternatively you can use PythonVirtualenvOperator and specify your pip packages in Operator.

Upvotes: 0

Jongwho Park
Jongwho Park

Reputation: 59

i used docker custom image.

FROM apache/airflow
COPY ./requirements.txt ./requirements.txt
RUN pip install -r ./requirements.txt

and push docker repository(i use aws ecr).

edit helm deployment file(like values.yaml)

images:
  airflow:
    repository: {docker image repository uri}
    tag: latest
    pullPolicy: IfNotPresent

Upvotes: 1

Related Questions