CODEWITHSUNDEEP
kubernetesairflowhashicorp-vault
Sadan A.
Sadan A.

Reputation: 1107

Airflow config from Vault secrets not working

I have an Airflow application running in kubernetes along with Vault. I am already using Vault to manage certain Airflow Connections and it's working fine (which proves the connectivity with Vault is working fine).

I am now trying to configure Airflow's configs core.sql_alchemy_conn, celery.result_backend and celery.broker_url through Vault as mentioned here.

In my values.yaml I have

secrets_backend_kwargs:
  auth_type: 'kubernetes'
  kubernetes_role: 'role-id'
  connections_path: 'applications/secrets/airflow/connections/'
  config_path: 'applications/secrets/airflow/config/'

config:
  core:
    sql_alchemy_conn_secret: "sql_alchemy_conn"
  celery:
    broker_url_secret: 'broker_url_conn'
    result_backend_secret: 'result_backend_conn'
  secrets:
    backend: 'airflow.providers.hashicorp.secrets.vault.VaultBackend'
    backend_kwargs: '{{ .Values.secrets_backend_kwargs | toJson }}'

In my vault, I have the proper encoded URI (postgresql%2Bpsycopg2%3A%2F%2Fblahblahblah...) stored at the right location:

enter image description here

Now when I deploy the application, run-airflow-migration job fails with AirflowConfigException while validating the configs.

# AirflowConfigException
error: cannot use sqlite with the CeleryExecutor

Which means it is not picking the postgres database connection settings from Vault and using sqlite settings from Airflow configs.

I also tried this by providing these configs in the variables for the containers.

env:
  - name: VAULT_ADDR
    value: 'https://path.to.vault/'
  - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET
    value: 'sql_alchemy_conn'
  - name: AIRFLOW__CELERY__BROKER_URL_SECRET
    value: 'broker_url_conn'
  - name: AIRFLOW__CELERY__RESULT_BACKEND_SECRET
    value: 'result_backend_conn'

I can see these environment variables in the container but the deployment fails because of the error in migration job.

I verified the Vault path and connectivity by changing the config_path in the secret backend kwargs and if I change that I still reach the Vault but since the key/connection id doesn't exist on the incorrect path so I get error from Vault. Which means that When I am setting the _secret as config, application is reaching Vault to fetch value but not updating or using the value.

I can't figure what am I missing.

Upvotes: 1

Views: 2929

Answers (1)

Sadan A.
Sadan A.

Reputation: 1107

Configs are set like Variables in the secret backend, while I was setting them like a Connection with conn_uri as key.

It is not clear in the official Documentation and had to go through the code to find the fix.

So after putting my configs in correct way (where mount point is secret and config_path is applications/secrets/airflow/config/), it worked.

vault kv put secret/applications/secrets/airflow/config/sql_alchemy_conn value=postgresql://user:pass@host:5432/db

Upvotes: 1

Related Questions