CODEWITHSUNDEEP
pythonairflowcondadirected-acyclic-graphs
tribo32
tribo32

Reputation: 335

How to change working directory and specify conda environment in Apache Airflow

I have a workflow I want to move into Airflow, the workflow is made up of multiple Python scripts and needs to be run in a Conda environment I have created.

At the minute I am using the BashOperator to run a shell script which activates my environment and then runs the python code, but this will get messy as I add more Python scripts to the workflow. At the minute this is what I am doing:

with dag:
    dictionary_generator = BashOperator(
        task_id='dictionary_generator',
        bash_command='bash -i /home/lnxuser/PycharmProjects/DEN/DEN_ns/dic.sh '
    )

the bash command runs the shell script 'dic.sh':

conda activate DEN_env3.9
cd /home/lnxuser/PycharmProjects/DEN/DEN_ns/
python3 Jbn/Ops/_dict_generation.py Job_File

This shell script activates the environment, changes the directory and then runs the python script.

This is not very neat, so I am looking for a way to specify the working directory airflow uses through the dag as well as a method to activate the environment (but this does not have to be specified in the dag).

I am relatively new to Airflow so any help or suggestions would be appreciated.

Upvotes: 2

Views: 2049

Answers (1)

Hans Bambel
Hans Bambel

Reputation: 956

I managed to it to work by specifying the python executable as mentioned here: https://stackoverflow.com/a/58440405

bash_command='/Users/<username>/opt/anaconda3/envs/DEN_env3.9/bin/python /path/to/python/script/my_script.py'

Upvotes: 1

Related Questions