CODEWITHSUNDEEP
airflow
Pierre
Pierre

Reputation: 988

Pass variable to template in Airflow

In Airflow I am creating branches with different operators with a for loop, my code looks like this:

for table in ['messages', 'conversations']:
    Operator1 with operator1.task_id = 'operator1_{}'.format(table)
    Operator1 does kwargs['ti'].xcom_push(key='file_name', value='y')

    Operator2 is a BashOperator that needs to run:
    bash_command = "echo {{ ti.xcom_pull(task_ids='operator1_{}', key='file_name') }}".format(table)       

    Operator1 >> Operator2

But in the UI the commands are rendered like that:

echo { ti.xcom_pull(task_ids='operator1_messages', key='file_name') }
echo { ti.xcom_pull(task_ids='operator1_conversations', key='file_name') }

How should I write the bash_command to have Airflow interpret correctly the template?

If I write directly

 bash_command = "echo {{ ti.xcom_pull(task_ids='operator1_messages', key='file_name') }}"

it works but I want to create this command from a for loop.

Thanks!

Upvotes: 0

Views: 2625

Answers (1)

Simon D
Simon D

Reputation: 6259

It's doing this because the .format(table) part of your bash command is stripping off the outer { and }. You may be able to fix this with the following instead:

bash_command = "echo {{ ti.xcom_pull(task_ids='operator1_" + table + "', key='file_name') }}"

Whether this is the best way to do it is probably another question.

Upvotes: 1

Related Questions