CODEWITHSUNDEEP
pythonjinja2airflow
pringlesss
pringlesss

Reputation: 21

How to get conf value from airflow dag?

I want to get conf value from dag area.

"{{ dag_run.conf['company'] }}" is recognized as a string.

How can I get this value?

Values ​​are passed fine when calling other dags.

t_trigger = TriggerDagRunOperator(
    task_id="t-trigger",
    trigger_dag_id="example_dag",
    conf={
        "company": "{{ dag_run.conf['company'] }}",
    },
)

However, in the dag area the value is recognized as a string.

t_task_a = PythonOperator(
    task_id="t-task-a",
    python_callable=task-a,
)
employees = Variable.get(
    "{{ dag_run.conf['company'] }}", # problem
    default_var=['company'],
    deserialize_json=True
)
for employee in employees:
    t_employee_operator = PythonOperator(
        task_id=f"t-test-operator",
        python_callable=employee_operator,
        op_kwargs={"employee": employee}
    )
    t_task_a >> t_employee_operator

Upvotes: 1

Views: 4642

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15911

As you already noticed Airflow does not render templates outside of Operators scope - this is expected. You can read this answer for more information about it.

Furthermore you are using Variable.get() as top level code which is very abusive. I explained the reasons in detail in this answer.

I would suggest to redesign your DAG structure to fit Airflow DAG writing practices.

Upvotes: 1

Related Questions