CODEWITHSUNDEEP
python-3.xdockergoogle-cloud-platform
ianyoung
ianyoung

Reputation: 3035

"OSError" whilst trying to run a Python app inside a Docker container using Application Default Credentials

Error

My Python app is running fine locally. I've created a Dockerfile and built an image. Inside the app I'm using the Python Google Cloud Logging library. When I try running the image I can see the following error from the Docker container logs:

File "/home/apprunner/app/./app/main.py", line 12, in <module>
2022-12-20 15:14:37     client = google.cloud.logging.Client()
2022-12-20 15:14:37   File "/home/apprunner/app/.venv/lib/python3.9/site-packages/google/cloud/logging_v2/client.py", line 122, in __init__
2022-12-20 15:14:37     super(Client, self).__init__(
2022-12-20 15:14:37   File "/home/apprunner/app/.venv/lib/python3.9/site-packages/google/cloud/client/__init__.py", line 320, in __init__
2022-12-20 15:14:37     _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
2022-12-20 15:14:37   File "/home/apprunner/app/.venv/lib/python3.9/site-packages/google/cloud/client/__init__.py", line 271, in __init__
2022-12-20 15:14:37     raise EnvironmentError(
2022-12-20 15:14:37 OSError: Project was not passed and could not be determined from the environment.

Running the Docker container

I'm running the Docker container using the following commands where I pass in Application Default Credentials:

# Set shell variable
ADC=~/.config/gcloud/application_default_credentials.json \

docker run \
-d \
-v ${ADC}:/tmp/keys/application_default_credentials.json:ro \
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/application_default_credentials.json \
IMAGE_NAME

I'm following the official guide on Docker with Google Cloud Access but using Application Default Credentials instead of a service account.

I've checked that my application_default_credentials.json is present in that location and I've checked that ${ADC} has the correct value:

$ echo $ADC
/Users/ian/.config/gcloud/application_default_credentials.json

Debugging

I see the stack trace points to the line in my code that calls the Logging library:

client = google.cloud.logging.Client()

And below it seems to suggest that it is expecting a project as well as the credentials:

_ClientProjectMixin.__init__(self, project=project, credentials=credentials)

Is this a problem with how I'm passing in my Application Default Credentials or should I be passing in some other project information?

Update

If I explicitly pass in the project argument in my code I can get the Docker container to run successfully:

client = google.cloud.logging.Client(project='my-project')

However I don't want to make code changes for local development and this shouldn't be required. I don't understand why this isn't be pulled out of my ADC(?)

Upvotes: 0

Views: 1251

Answers (1)

ianyoung
ianyoung

Reputation: 3035

I've been able to get it to run but only by explicitly passing in the project ID.

Solution

The GOOGLE_CLOUD_PROJECT variable is an explicit requirement alongside GOOGLE_APPLICATION_CREDENTIALS. The cleanest way is to pass both in as environment variables when running the container. This is the first place that is searched.

Set shell vars:

ADC=~/.config/gcloud/application_default_credentials.json \
PROJECT=my-project

Docker run:

$ docker run \
-v ${ADC}:/tmp/keys/application_default_credentials.json:ro \
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/application_default_credentials.json \
-e GOOGLE_CLOUD_PROJECT=${PROJECT} \
IMAGE_NAME

Explanation

The docs mention that the project should be inferred from the environment if not explicitly provided:

# if project not given, it will be inferred from the environment
client = google.cloud.logging.Client(project="my-project")

To be inferred from the environment you need to have:

  1. Installed the Google Cloud SDK
  2. Created Application Default Credentials (ADC).
    gcloud auth application-default login
  3. Set an active project.
    gcloud config set project PROJECT_ID

Passing ADC in to the locally running container through environment variables works for authentication but it doesn't pass in the active project as this is set in your local configuration (3)(~/.config/gcloud/configurations on Mac/Linux). So no project can be inferred from the environment inside the container as it is not set and not passed. So it searches through the list of locations in order and doesn't find anything.

Best Practice

It's good practice to pass both authentication credentials and project identifier from the same place:

Credentials and project information must come from the same place (principle of least surprise).

Be explicit in setting them rather than relying on Application Default Credentials:

we really encourage people to explicitly pass credentials and project to the client constructor and not depend on the often surprising behavior of application default credentials.

Make the easy to find by setting them in the first place that will be searched:

  1. GOOGLE_CLOUD_PROJECT environment variable
  2. GOOGLE_APPLICATION_CREDENTIALS JSON file

With these in mind, passing them both in as environment variables ticks all the boxes.

Note: This looks to be the same throughout the Google Cloud Python Client Library and not just in Logging.

Upvotes: 2

Related Questions