CODEWITHSUNDEEP
airflowgoogle-cloud-composer
Datageek
Datageek

Reputation: 26679

Using kubernetes-secrets with Google Composer

Is it possible to use kubernetes-secrets together with Google Composer in order to access secrets from Airflow workers?

We are using k8s secrets with our existing standalone k8s Airflow cluster and were hoping we can achieve the same with Google Composer.

Upvotes: 1

Views: 1122

Answers (2)

hexacyanide
hexacyanide

Reputation: 91599

By default, Kubernetes secrets are not exposed to the Airflow workers deployed by Cloud Composer. You can patch the deployments to add them (airflow-worker and airflow-scheduler), but there will be no guarantee that they won't be reverted if you perform an update on the environment (such as configuration update or in-place upgrade).

It's probably easiest to use an Airflow connection (which are encrypted in the metadata database using Fernet), or to launch new pods using KubernetesPodOperator/GKEPodOperator and mounting the relevant secrets into the pod at pod launch.

Upvotes: 1

aga
aga

Reputation: 3883

Kubernetes secrets are available to the Airflow workers. You can contribute the components for whatever API you wish to call to work natively in Airflow so that the credentials can be stored as a Connection in Airflow's metadata database, which is encrypted at rest. Using Airflow connection involves storing the secret key in GCS with an appropriate ACL, and setting up Composer to secure the connection.

You can write your own custom operator to access the secret in the Kubernetes and use it. Take a look for SimpleHttpOperator - this pattern can be applied to any arbitrary secret management scheme. This is for for scenarios that access external services that aren't explicitly supported by Airflow Connections, Hooks, and Operators.

I hope it helps.

Upvotes: 1

Related Questions