CODEWITHSUNDEEP
airflow
Gabe
Gabe

Reputation: 6055

Send failure email to recipients based on variable using Jinja

I'm trying to send an email to certain people defined in a variable in the Airflow UI. I see a similar question here but mine is slightly different. I suspect I can't do this when I'm using Jinja templating?

Apache airflow sends sla miss emails only to first person on the list

My variable:

{ "config1" : "abc", "config2":500000, "email_on_failure_list": ["[email protected]","[email protected]"]}

My dag. it's bad practice to use Variable.Get() in top level code when referencing a variable created in the Airflow UI. so I'm using Jinja templating but it doesn't seem to be working

email = "{{ var.json.my_dag.email_on_failure_list}}"

default_args = {
    'depends_on_past': False,
    'start_date': datetime(2020, 9, 1),
    'email_on_failure': True,
    'email': email 
}

this is the error I receive

 ERROR - Failed to send email to: {{ var.json.my_dag.email_on_failure_list}}
[2021-02-09 09:13:16,339] {{taskinstance.py:1201}} ERROR - {'{{ var.json.my_dag.email_on_failure_list}}': (501, b'5.1.3 Invalid address')}

Upvotes: 1

Views: 912

Answers (1)

Gabe
Gabe

Reputation: 6055

Short answer: it's not possible with Jinja templates.

The alternative that was suggested on the Airflow Slack channel (Thanks Kaxil!) is to create an Environment Variable. I can then reference this using Variable.Get without any of the performance issues.

Environment variables aren't ideal for something non-sensitive since it requires more access to modify but it's a decent alternative to have different configuration settings across different Airflow environments (dev, qa, prod).

Follow the instructions here: https://airflow.apache.org/docs/apache-airflow/2.0.1/howto/variable.html#storing-variables-in-environment-variables

Steps:

  1. Create an environment variable on the airflow server (ex. AIRFLOW_VAR_MY_EMAIL_LIST). It's important to not create the variable in the Airflow UI

  2. use it in your dag

from airflow.models import Variable
my_email_list = Variable.get("AIRFLOW_VAR_MY_EMAIL_LIST")
    
default_args = {
        'depends_on_past': False,
        'start_date': datetime(2020, 9, 1),
        'email_on_failure': True,
        'email': my_email_list 
}

Upvotes: 1

Related Questions