CODEWITHSUNDEEP
pythongoogle-cloud-platformairflow
Oladayo
Oladayo

Reputation: 23

Specify GOOGLE APPLICATION CREDENTIALS in Airflow

So I am trying to orchestrate a workflow in Airflow. One task is to read GCP Cloud Storage, which needs me to specify the Google Application Credentials.

I decided to create a new folder in the dag folder and put the JSON key. Then I specified this in the dag.py file;

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "dags\support\keys\key.json"

Unfortunately, I am getting this error below;

google.auth.exceptions.DefaultCredentialsError: File dags\support\keys\dummy-surveillance-project-6915f229d012.json was not found

Can anyone help with how I should go about declaring the service account key?

Thank you.

Upvotes: 1

Views: 4250

Answers (2)

Mazlum Tosun
Mazlum Tosun

Reputation: 6572

You can create a connection to Google Cloud from Airflow webserver admin menu. In this menu you can pass the Service Account key file path.

enter image description here

In this picture, the keyfile Path is /usr/local/airflow/dags/gcp.json.

Beforehand you need to mount your key file as a volume in your Docker container with the previous path.

You can also directly copy the key json content in the Airflow connection, in the keyfile Json field :

enter image description here

You can check from these following links :

Airflow-connections

Airflow-with-google-cloud

Airflow-composer-managing-connections

Upvotes: 3

Simon D
Simon D

Reputation: 6259

If you trying to download data from Google Cloud Storage using Airflow, you should use the GCSToLocalFilesystemOperator operator described here. It is already provided as part of the standard Airflow library (if you installed it) so you don't have to write the code yourself using the Python operator.

Also, if you use this operator you can enter the GCP credentials into the connections screen (where it should be). This is a better approach to putting your credentials in a folder with your DAGs as this could lead to your credentials being committed into your version control system which could lead to security issues.

Upvotes: 0

Related Questions