CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-bigquerygoogle-cloud-vertex-ai
RobM
RobM

Reputation: 21

Remove GOOGLE_APPLICATION_CREDENTIALS from python environment and revert to Google SDK default location

I am trying to run python and jupyter scripts locally, connecting to GCP (specifically BigQuery and VertexAI for data science and llm purposes)

Originally, I had trouble setting this up, so at some point, I tried to create a service account key and set my GOOGLE_APPLICATION_CREDENTIALS to a stored json in order to connect. (following this documentation: https://cloud.google.com/docs/authentication/provide-credentials-adc#how-to) I never managed to get this to work.

I have now managed to use the preferred method of connecting in the Google Cloud SDK, and can authenticate for my project.

(what I mean is I can run the following and authenticate via the browser and access the project information via the Google SDK in cmd: gcloud auth application-default login or gcloud auth application-default login --project=(project name) )

However, when I am in python, it is still trying to use the stored json, rather than the default connection path.

The following code fails:

from google.cloud import bigquery
import os

print(os.environ['GOOGLE_APPLICATION_CREDENTIALS'])

>> (old json path)

client = bigquery.Client()

>> DefaultCredentialsError: File (old json path) was not found

The following works:

from google.cloud import bigquery
import os 

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] ='C:/Users/(user)/AppData/Roaming/gcloud/application_default_credentials.json'

client = bigquery.Client(project=(project name))

However when I restart the kernel, os.environ['GOOGLE_APPLICATION_CREDENTIALS'] reverts to the old json path.

Within the Google SDK, I have tried clearing and resetting the GOOGLE_APPLICATION_CREDENTIALS like this:

GOOGLE_APPLICATION_CREDENTIALS=

(trying to set it to nothing to try and clear it)

and

GOOGLE_APPLICATION_CREDENTIALS='C:/Users/(user)/AppData/Roaming/gcloud/application_default_credentials.json'

(trying to force it to use the default path)

While these work within the SDK, (I can connect to GCP, see all my projects, etc.) These changes do not reflect within the python environment.

As additional information, I could use the temporary fix, however I am pulling code from a repo which I would need to edit before running each time, and then remove before pushing changes back. I have also tried recreating my venv, in case the json path is stored in the venv somewhere.

How can I get python to always use the default credentials from the SDK, and remove the reference to the stored json?

(this is my first post, so I hope I am being clear enough)

edit to add: I have also tried using the Google Python library (which I don't understand very well) within the notebook to run:

import google.auth
credentials, project = google.auth.default()

and get the same error: DefaultCredentialsError: File (old json path) was not found

I am mainly following this documentation: Google Cloud Client Libraries - API-Core

Upvotes: 1

Views: 555

Answers (1)

RobM
RobM

Reputation: 21

Ok, I figured it out and it was way easier than I was expecting... At some stage I had set the environment variable to the old json path containing the service account key. I thought the the Google SDK would over write this when I changed it, which it was not doing.

To fix it, I just went to the Windows Environment Variables manager in Control Panel and deleted the GOOGLE_APPLICATION_CREDENTIALS variable.

It now uses the default credentials as expected.

Upvotes: 1

Related Questions