CODEWITHSUNDEEP
airflow
Ian
Ian

Reputation: 337

How to use requirements.txt file in Airflow's PythonVirtualenvOperator?

In Airflow 2.2.3, PythonVirtualenvOperator was updated to allow templated requirements.txt files in the requirements parameter. However, I am unable to properly utilize that parameter. I am seeking guidance on how to properly utilize a requirements.txt file with Airflow's PythonVirtualenvOperator.

Here's my /dags format:

dags
├── .airflowignore
├── daily.py
└── modules
    └── monday
        ├── monday.py
        └── requirements.txt

In daily.py, I define my daily DAG. The task I'd like to reference requirements.txt for is defined like so:

@task.virtualenv(requirements='modules/monday/requirements.txt')
def sync_board_items():
    from modules.monday.monday import sync_board_items
    import logging
    logging.basicConfig(level=logging.INFO)
    sync_board_items(board_id=XXXX, table=XXXX)

This seems to fit the implementation described in GitHub, because requirements is a string, not a list, and it complies with the *.txt template. However, when the task runs, I quickly receive an error:

Executing cmd: /tmp/venvfn63dy3c/bin/pip install m o d u l e s / m o n d a y / r e q u i r e m e n t s . t x t
ERROR: Directory '/' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

This seems to indicate that PythonVirtualenvOperator is treating my requirements param like a list instead of a string. In other words, I am doing something wrong and the PythonVirtualenvOperator is not properly handling my requirements.txt file. What am I missing or how can I leverage a requirements.txt file in Airflow's PythonVirtualenvOperator?

Upvotes: 2

Views: 5482

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15961

The feature is available only for Airflow>=2.3.0 it's not available in 2.2.3

Currently 2.3.0 is under testing. 2.3.0b1 is available via pip.

if you want to use this feature in 2.2.3 you will need to create a custom operator by backporting the code in the PR. Creating a MyPythonVirtualenvOperator with the code in the PR and utils should work fine in 2.2.3

Upvotes: 3

Related Questions