Google Python SDK : google.auth.exceptions.RefreshError: ('No access token in response.',

Question

I am trying to make basic calls for gsuite, ala listing groups. I have poked around quite a bit and have seen several posts suggesting this is a scope issue. However, I have included as many scope paths as I can find that even looked remotely relevant. I still get an exception in the library because there is no "access_token" in the response. There IS an id_token, but nothing labelled "access_token", so the lib throws an exception.

Also, I am pretty sure the account has access to the "group settings API", and many other things.

Banging my head on the desk...

google-api-core==2.10.1
google-api-python-client==2.64.0
google-auth==2.12.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.5.3
googleapis-common-protos==1.56.4

from __future__ import print_function

import os.path

from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from google.oauth2 import service_account

SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin',
          'https://www.googleapis.com/auth/admin.directory.user',
          "https://www.googleapis.com/auth/apps.groups.settings",
          'https://www.googleapis.com/auth/admin',
          'https://www.googleapis.com/auth/apps.groups.settings',
          'https://www.googleapis.com/auth/indexing',
          'https://www.googleapis.com/auth/cloud-platform',
          'https://www.googleapis.com/auth/contacts',
          'https://www.googleapis.com/auth/contacts.other.readonly'
          'https://www.googleapis.com/auth/contacts.readonly',
          'https://www.googleapis.com/auth/directory.readonly',
          'https://www.googleapis.com/auth/user.addresses.read',
          'https://www.googleapis.com/auth/user.birthday.read',
          'https://www.googleapis.com/auth/user.emails.read',
          'https://www.googleapis.com/auth/user.gender.read',
          'https://www.googleapis.com/auth/user.organization.read',
          'https://www.googleapis.com/auth/user.phonenumbers.read',
          'https://www.googleapis.com/auth/userinfo.email',
          'https://www.googleapis.com/auth/userinfo.profile',
          'https://www.googleapis.com/auth/userinfo.email',
          'https://www.googleapis.com/auth/admin',
          'https://www.googleapis.com/auth/service.management.readonly',
          'https://www.googleapis.com/auth/monitoring',
          'https://www.googleapis.com/auth/admin.reports.audit.readonly',
          'https://www.googleapis.com/auth/admin.reports.usage.readonly',
          'https://www.googleapis.com/auth/admin.datatransfer',
          'https://www.googleapis.com/auth/admin.datatransfer.readonly',
          'https://www.googleapis.com/auth/admin.directory.customer',
          'https://www.googleapis.com/auth/admin.directory.customer.readonly',
          'https://www.googleapis.com/auth/admin.directory.domain',
          'https://www.googleapis.com/auth/admin.directory.domain.readonly',
          'https://www.googleapis.com/auth/admin.directory.group',
          'https://www.googleapis.com/auth/admin.directory.group.member',
          'https://www.googleapis.com/auth/admin.directory.group.member.readonly',
          'https://www.googleapis.com/auth/admin.directory.group.readonly',
          'https://www.googleapis.com/auth/admin.directory.orgunit',
          'https://www.googleapis.com/auth/admin.directory.orgunit.readonly',
          'https://www.googleapis.com/auth/admin.directory.resource.calendar',
          'https://www.googleapis.com/auth/admin.directory.resource.calendar.readonly',
          'https://www.googleapis.com/auth/admin.directory.rolemanagement',
          'https://www.googleapis.com/auth/admin.directory.rolemanagement.readonly',
          'https://www.googleapis.com/auth/admin.directory.user',
          'https://www.googleapis.com/auth/admin.directory.user.alias',
          'https://www.googleapis.com/auth/admin.directory.user.alias.readonly',
          'https://www.googleapis.com/auth/admin.directory.user.readonly',
          'https://www.googleapis.com/auth/admin.directory.user.security',
          'https://www.googleapis.com/auth/admin.directory.userschema',
          'https://www.googleapis.com/auth/admin.directory.userschema.readonly',
          'https://www.googleapis.com/auth/groups',
          'https://www.googleapis.com/auth/userinfo.email',
          'https://www.googleapis.com/auth/cloud-platform',
          'https://www.googleapis.com/auth/cloud-platform.read-only',
          'https://www.googleapis.com/auth/apps.groups.migration']
SERVICE_ACCOUNT_FILE = 'privkey.json'

def main():

    credentials = service_account.Credentials.from_service_account_file(
                SERVICE_ACCOUNT_FILE, scopes=SCOPES)

    
    service = build('admin', 'directory_v1', credentials=credentials)

    # this is where the library throws the exception
    foobar = service.groups().list().execute()

if __name__ == '__main__':
    main()

This is a partial stack trace:

Traceback (most recent call last):
  File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/oauth2/_client.py", line 294, in jwt_grant
    access_token = response_data["access_token"]
KeyError: 'access_token'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../gsuite_auth_test6.py", line 93, in <module>
    main()
  File "../gsuite_auth_test6.py", line 90, in main
    foobar = service.groups().list().execute()
  File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/XXXXXXXXXX/.local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
    headers=self.headers,
  File "/XXXXXXXXXXXXX/.local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google_auth_httplib2.py", line 209, in request
    self.credentials.before_request(self._request, method, uri, request_headers)
  File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/oauth2/service_account.py", line 411, in refresh
    request, self._token_uri, assertion
  File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/oauth2/_client.py", line 299, in jwt_grant
    six.raise_from(new_exc, caught_exc)
  File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ('No access token in response.', {'id_token': '....'

Answer 1

Your research is right in that this can be caused by wrong scopes but there are more issues here that I will try to address.

I have included as many scope paths as I can find that even looked remotely relevant.

This is a problem. You cannot just throw all scopes to the wall and see what sticks. First you need to have enabled the relevant APIs on your GCP project. In this case you don't actually need the Groups Settings API, instead you need the Directory API, also known as Admin SDK API:

Admin SDK lets administrators of enterprise domains to view and manage resources like user, groups etc. It also provides audit and usage reports of domain.

Next, to know the scopes you can refer to Google's documentation. The groups.list method has the scopes that you need to use to list all Groups in the domain. In this case you can just use https://www.googleapis.com/auth/admin.directory.group.

Also, in the groups.list documentation I linked above you can see that you need to specify either the parameter customer for all groups within the Google Workspace account or domain for just one of its subdomains. You can also just use the my_customer alias to search everything within the account. The Python documentation here has its equivalent for the Python library. All of this means that your line to call groups.list should specify the customer like this:

foobar = service.groups().list(customer="my_customer").execute()

Finally, you seem to be using a service account's credentials. This means that you need to first enable domain-wide delegation for your service account so it can impersonate your user accounts, next, you have to impersonate an admin that already has the privileges to manage groups in the Admin Console. You can refer to the Google OAuth docs to check out how to do this, and here's also a Python sample. Pretty much you can just create a copy of the credentials while specifying the admin account, like this:

delegated_credentials=credentials.with_subject("admin@yourdomain.com")

Here's a sample based on your code that worked for me after taking all of the above into account:

from __future__ import print_function

import os.path

from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from google.oauth2 import service_account

SCOPES = ['https://www.googleapis.com/auth/admin.directory.group']

SERVICE_ACCOUNT_FILE = 'privkey.json'

def main():

    credentials = service_account.Credentials.from_service_account_file(
                SERVICE_ACCOUNT_FILE, scopes=SCOPES)

    #delegate the admin credentials to the service account
    delegated_credentials=credentials.with_subject("admin@yourdomain.com")
    
    service = build('admin', 'directory_v1', credentials=delegated_credentials)

    # make sure to add the customer parameter per the documentation requirements
    foobar = service.groups().list(customer="my_customer").execute()

    print(foobar)

if __name__ == '__main__':
    main()

As you can see, aside from removing the unnecessary scopes I just changed two lines. First make sure that your GCP and service account configurations are in order and then don't forget to delegate access and specify the customer variable.