GAE AttributeError: 'Credentials' object has no attribute 'with_subject'

Question

I have a python app I want to deploy on App Engine (2nd Generation Python 3.7) on which I use a Service Account with Domain-wide delegation enabled to access user data.

Locally I do:

import google.auth
from apiclient.discovery import build

creds, project = google.auth.default(
    scopes=['https://www.googleapis.com/auth/admin.directory.user', ],
)
creds = creds.with_subject(GSUITE_ADMIN_USER)

service = build('admin', 'directory_v1', credentials=creds)

This works good and, as far as I know it is the current way to do this when using Application Default Credentials (locally I have GOOGLE_APPLICATION_CREDENTIALS defined).

Problem is on GAE, when deployed, the call to with_subject raises: AttributeError: 'Credentials' object has no attribute 'with_subject'

I have enabled Domain-wide delegation on the GAE service account already.

What is different between the GOOGLE_APPLICATION_CREDENTIALS I use locally and the ones in GAE when both are service accounts with domain-wide delegation?

Where is .with_subject() on GAE?

The creds object received is of type compute_engine.credentials.Credentials.

Full traceback:

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/env/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 104, in init_process
    super(ThreadWorker, self).init_process()
  File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/env/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/env/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/srv/main.py", line 1, in <module>
    from config.wsgi import application
  File "/srv/config/wsgi.py", line 38, in <module>
    call_command('gsuite_sync_users')
  File "/env/lib/python3.7/site-packages/django/core/management/__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "/env/lib/python3.7/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/srv/metanube_i4/users/management/commands/gsuite_sync_users.py", line 14, in handle
    gsuite_sync_users()
  File "/env/lib/python3.7/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/env/lib/python3.7/site-packages/celery/app/task.py", line 375, in __call__
    return self.run(*args, **kwargs)
  File "/srv/metanube_i4/users/tasks.py", line 22, in gsuite_sync_users
    creds = creds.with_subject(settings.GSUITE_ADMIN_USER)
AttributeError: 'Credentials' object has no attribute 'with_subject'"  

Packages (partial list):

google-api-core==1.5.0
google-api-python-client==1.7.4
google-auth==1.5.1
google-auth-httplib2==0.0.3
google-cloud-bigquery==1.6.0
google-cloud-core==0.28.1
google-cloud-logging==1.8.0
google-cloud-storage==1.13.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
httplib2==0.11.3
oauthlib==2.1.0

Answer 1

I was in a similar situation trying to run a query on a federated BQ table linked to a Google Sheet via a Cron job in GAE v2. The Google Drive scope which is required was not available to the default service account in GAE. vkopio's answer is great and I ended up using it as well because it looks cleaner, but here's another solution which doesn't require the Service Account Token Creator role to be assigned to the service account. I put it together while combing through the documentation for Cloud Functions (which uses underlying compute architecture similar to GAE) using the Rest API.

import requests

METADATA_URL = 'http://metadata.google.internal/computeMetadata/v1'
METADATA_HEADERS = {'Metadata-Flavor': 'Google'}
SERVICE_ACCOUNT = 'default'
SCOPES=['https://www.googleapis.com/auth/admin.directory.user']


def get_access_token(scopes):
    """
    Retrieves an access_token in App Engine for the default service account

    :param scopes: List of Google scopes as strings
    :return: access token as string
    """
    scopes_str = ','.join(scopes)
    url = f'{METADATA_URL}/instance/service-accounts/{SERVICE_ACCOUNT}/token?scopes={scopes_str}'
    # Request an access token from the metadata server.
    r = requests.get(url, headers=METADATA_HEADERS)
    r.raise_for_status()
    # Extract the access token from the response.
    access_token = r.json()['access_token']
    return access_token

I was able to use this access_token in my header for my request

headers = {'Authorization': f'Bearer {access_token}'}

r = requests.post(url, json=job_body, headers=headers)

where url points to the specific Rest endpoint I want to call with the appropriate configuration in job_body. Note that this does not work outside of the App Engine environment.

There was a way to create credentials using AccessTokenCredentials in oauth2client but it is now deprecated by Google, so this method requires using the Rest endpoints directly. Posting this answer so it's helpful for others who might not want to add any additional roles to the Service Account.

Answer 2

It is true that you cannot use the with_subject method with GAE or GCE credentials. However, there is a workaround that I was able to get working on my GCE server and I would assume this works with GAE default service accounts as well. The solution is to build new credentials using the service account identity with desired subject and scopes. A detailed guide can be found here, but I will also explain the process bellow.

Firstly, the service account needs permissions to create service account tokens for itself. This can be done by going to the projects IAM and admin > Service accounts page (make sure the info panel is visible, it can be toggled from the top right corner). Copy the service account email address and select the service account in question by ticking the checkbox. Now the info panel should have ADD MEMBER button. Click it and paste the service account email to the New members textbox. Click the Select role dropdown and choose the role Service Accounts -> Service Account Token Creator. You can check that the role is assigned with the following gcloud command:

gcloud iam service-accounts get-iam-policy [SERVICE_ACCOUNT_EMAIL]

Now to the actual Python code. This example is a slight modification from the documentation linked above.

from googleapiclient.discovery import build
from google.auth import default, iam
from google.auth.transport import requests
from google.oauth2 import service_account

TOKEN_URI = 'https://accounts.google.com/o/oauth2/token'
SCOPES = ['https://www.googleapis.com/auth/admin.directory.user']
GSUITE_ADMIN_USER = 'admin@example.com'

def delegated_credentials(credentials, subject, scopes):
    try:
        # If we are using service account credentials from json file
        # this will work
        updated_credentials = credentials.with_subject(subject).with_scopes(scopes)
    except AttributeError:
        # This exception is raised if we are using GCE default credentials

        request = requests.Request()

        # Refresh the default credentials. This ensures that the information
        # about this account, notably the email, is populated.
        credentials.refresh(request)

        # Create an IAM signer using the default credentials.
        signer = iam.Signer(
            request,
            credentials,
            credentials.service_account_email
        )

        # Create OAuth 2.0 Service Account credentials using the IAM-based
        # signer and the bootstrap_credential's service account email.
        updated_credentials = service_account.Credentials(
            signer,
            credentials.service_account_email,
            TOKEN_URI,
            scopes=scopes,
            subject=subject
        )
    except Exception:
        raise

    return updated_credentials


creds, project = default()
creds = delegated_credentials(creds, GSUITE_ADMIN_USER, SCOPES) 

service = build('admin', 'directory_v1', credentials=creds)

The try block will not fail if you have GOOGLE_APPLICATION_CREDENTIALS environment variable set with a path to a service account file. If the application is run on Google Cloud, there will be an AttributeError and it is handled by creating new credentials which have correct subject and scopes.

You can also pass None as the subject for delegated_credentials function and it creates the credentials without delegation so this function can be used with or without delegation.

Answer 3

@marc.fargas You may have a look at the googleapis/google-auth-library-python library on GitHub. You'll find some information relevant to the method in question:

The credentials are considered immutable. If you want to modify the scopes or the subject used for delegation, use :meth:with_scopes or :meth:with_subject:: scoped_credentials = credentials.with_scopes(['email']) delegated_credentials = credentials.with_subject(subject)

As you defined your Application Default Credentials with "GOOGLE_APPLICATION_CREDENTIALS", you were getting an instance of google.auth.service_account.Credentials which has the with_subject method.

While on App Engine, you are instead getting an instance of app_engine.Credentials, which does not have the with_subject method. This explains the observed behavior and the error you see.

According to the documentation on Domain-wide delegation, only Service account credentials can have domain-wide delegation.