CODEWITHSUNDEEP
airflow
Blessy
Blessy

Reputation: 520

Difference between Airflow environment variable and secret environment variable

What is the difference between Airflow environment variable (Eg. AIRFLOW__CORE__SQL_ALCHEMY_CONN) and a secret environment variable (AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET)?

Upvotes: 0

Views: 1322

Answers (2)

kaxil
kaxil

Reputation: 18894

You can set Airflow configs containing Sensitive data using either the command to run passed via Environment Variable (AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD) or using Airflow Secrets (e.g AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET).

For example, if you are using Hashicorp Vault as Airflow Secert's Backend (docs):

You can add the secret value in Vault as follows:

❯ vault kv put secret/config/sql_alchemy_conn value=sqlite:////Users/airflow/airflow/airflow.db
Key              Value
---              -----
created_time     2020-11-20T20:05:21.517705Z
deletion_time    n/a
destroyed        false
version          1

and set the following environment Variables:

❯ env | grep AIRFLOW
AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
AIRFLOW__SECRETS__BACKEND=airflow.contrib.secrets.hashicorp_vault.VaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"url":"http://127.0.0.1:8200","auth_type":"token", "token": "s.OddaHiiDdddpW6gNnqtd2lJ"}'

This way you don't need to expose your secrets in airflow.cfg file.

Upvotes: 1

faisal
faisal

Reputation: 71

AIRFLOW__CORE__SQL_ALCHEMY_CONN You are supplying the connection information directly.

AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET This allows you to provide the name of a secret which contains your connection information.

This will retrieve config option from Secret Backends e.g Hashicorp Vault. See Secrets Backends for more details.

https://airflow.readthedocs.io/en/stable/howto/set-config.html

Upvotes: 1

Related Questions