CODEWITHSUNDEEP
pythonsqlairflow
Pierre
Pierre

Reputation: 988

Airflow - Jinja template inside params (postgresoperator)

I am using many Postgres operators inside my Airflow DAG (building it with a for loop) and I wanted to know if it's possible to pass a JINJA template to params to my operator like: 

params = {'max_field': '{{ ti.xcom_pull(task_ids="get_max_field_' + table + '", key="max_field") }}'}

so that in the .sql file I would only need to have a query that looks like

SELECT .... FROM .... WHERE ... > '{{ params.max_field }}'

My problem is that I have so many queries now that doing directly

SELECT .... FROM .... WHERE ... > '{{ ti.xcom_pull(task_ids="get_max_field_table1'", key="max_field") }}'

has a lot of risks of errors (writing table2 instead of table1 for example when people do copy pastes).

I tried putting params in template_fields = ('sql', 'parameters', 'params') but it didn't work, it renders my query with '{{ ti.xcom_pull(task_ids="get_max_field_table1'", key="max_field") }}' instead of the value.

I also tried using parameters instead of params and then doing:

SELECT .... FROM .... WHERE ... > '%(max_field)s'

but then there are problems in my queries if there is a % in the query like:

SELECT .... FROM .... WHERE ... > '%(max_field)s' and text like '%hello%'

Upvotes: 3

Views: 2184

Answers (1)

kaxil
kaxil

Reputation: 18844

params doesn't support Jinja template.

Upvotes: 3

Related Questions