CODEWITHSUNDEEP
dockerairflow
Yannick Widmer
Yannick Widmer

Reputation: 1363

use airflow variables in BashOperator dag

I try to install the python requirements with following Dag

import airflow
from datetime import datetime, timedelta
from airflow.operators.bash_operator import BashOperator
import logging


args = {
    'owner': 'airflow',
    'start_date': airflow.utils.dates.days_ago(1),
    'provide_context': True
}


dag = airflow.DAG(
    'set_up_python',
    schedule_interval="@once",
    default_args=args,
    max_active_runs=1)

t1 = BashOperator(
        task_id= 'echo_varconf_path',
        bash_command="echo {{ var.conf_path }}",
        dag=dag
   )

t2 = BashOperator(
        task_id= 'install_python_requirements',
        bash_command="pip install {{ var.conf_path }}/requirements.txt",
        dag=dag
   )

Prior I set the variable conf_path however I can't reach the variable in the bash command. In the logs it seems that the {{ var.conf_path }} part is simply ignored:

[2018-03-29 23:56:25,858] {{base_task_runner.py:98}} INFO - Subtask: [2018-03-29 23:56:25,858] {{bash_operator.py:88}} INFO - Running command: pip install /requirements.txt [2018-03-29 23:56:25,866] {{base_task_runner.py:98}} INFO - Subtask: [2018-03-29 23:56:25,866] {{bash_operator.py:97}} INFO - Output: [2018-03-29 23:56:26,688] {{base_task_runner.py:98}} INFO - Subtask: [2018-03-29 23:56:26,687] {{bash_operator.py:101}} INFO - Invalid requirement: '/requirements.txt'

As you see in my dag definition I try to output the variable in a first task, the output of this one is empty too:

[2018-03-29 23:56:25,856] {{base_task_runner.py:98}} INFO - Subtask: [2018-03-29 23:56:25,856] {{bash_operator.py:88}} INFO - Running command: echo [2018-03-29 23:56:25,866] {{base_task_runner.py:98}} INFO - Subtask: [2018-03-29 23:56:25,865] {{bash_operator.py:97}} INFO - Output:

Upvotes: 1

Views: 6111

Answers (1)

joebeeson
joebeeson

Reputation: 4366

Try using {{ var.value.conf_path }} -- that should get you to the VariableAccessor instance in the template context, here.

Upvotes: 1

Related Questions