CODEWITHSUNDEEP
pythonairflow
Carlo
Carlo

Reputation: 441

Extract Keyfile JSON from saved connection of type "google_cloud_platform"

I have saved a connection of type "google_cloud_platform" in Airflow as described here https://cloud.google.com/composer/docs/how-to/managing/connections

Now in my DAG, I need to extract from the saved connection the Keyfile JSON

What is the correct hook to be used?

Upvotes: 0

Views: 3687

Answers (3)

valkmit
valkmit

Reputation: 78

The other solutions no longer work. Here's a way that's working in 2023:

from airflow.models import Connection

conn = Connection.get_connection_from_secrets(
    conn_id='my-gcp-connection'
)

json_key = conn.extra_dejson['keyfile_dict']

with open('gcp_svc_acc.json', 'w') as f:
    f.write(json_key)

Mostly because the imports moved around I think.

The question specifically refers to keyfile JSON, but this is a quick addendum for those who configured keyfile path instead: take care to check if it's keyfile_dict OR keyfile_path that the Airflow admin configured, as they're two different ways to set up the connection.

Upvotes: 2

Oluwafemi Sule
Oluwafemi Sule

Reputation: 38982

Use airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook to get the stored connection. For example

from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook

gcp_hook = GoogleCloudBaseHook(gcp_conn_id="<your-conn-id>")
keyfile_dict = gcp_hook._get_field('keyfile_dict')

Upvotes: 3

kaxil
kaxil

Reputation: 18874

You can just use BaseHook as follows:

from airflow.hooks.base_hook import BaseHook

GCP_CONNECTION_ID="my-gcp-connection"
BaseHook.get_connection(GCP_CONNECTION_ID).extras["extra__google_cloud_platform__keyfile_dict"]

Upvotes: 1

Related Questions