CODEWITHSUNDEEP
airflow
nick_rinaldi
nick_rinaldi

Reputation: 707

Is there any benefit to setting up an airflow connection on the UI, as opposed to on the CLI, or as ENV variables?

Question is in the title. I'm looking to set up hooks that connect to my external connections in Airflow. I'm wondering if there's any benefit to setting up the connection in ENV variables locally, rather than having to connection to the UI connection I set up.

Thanks!

Upvotes: 0

Views: 323

Answers (1)

Bas Harenslak
Bas Harenslak

Reputation: 3084

Each comes with pros and cons:

  • Setting connections via the UI is a manual process. Connections are stored encrypted (if fernet key is set) in the Airflow database.
  • Setting connections via the CLI can help automating connection creation. Connections are also stored encrypted (if fernet key is set) in the Airflow database.
  • Setting connections via environment variables is often done in containerized deployments (Docker Compose/Kubernetes) and you can keep connections outside of Airflow whilst tearing down/creating Airflow deployments. Note that connections set via env vars are not visible in the Airflow UI, you will be able to use them in Airflow code though.
  • A 4th option you might consider is a "secrets backend" (docs) where you can store and fetch connections from a supported secret management system such HashiCorp Vault. This allows you to leverage a system purpose-built for saving secret, which can often also be used by other software.

Upvotes: 3

Related Questions