CODEWITHSUNDEEP
jinja2airflow
Newskooler
Newskooler

Reputation: 4255

How to execute python code and airflow macro in a BashOperator?

I am trying to execute the following task in Airflow:

time_zone = "America/New_York"
t1= BashOperator(
    task_id='example_task',
    bash_command=('script.sh --arg1 hello '
                  f'--arg2 {{ execution_date + timedelta(days=1).astimezone(pytz.timezone({time_zone})) }}'),
    dag=dag)

The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.

My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.

Upvotes: 0

Views: 1364

Answers (1)

Newskooler
Newskooler

Reputation: 4255

The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.

There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.

It's not the cleanest solution, but it works.

Upvotes: 2

Related Questions