GMAIL imap access with OAuth2 and service account

Question

I have the following scenario:

• A server that runs a scheduled service reading a mailbox to handle attached files in new mails
• Its on a gmail account

Until now this worked with App Passwords (which will be switched off).

I'm able to create the token and login to imap(ssl) for a single user (with web login), but for a service account there seems to be no way to "impersonate" the user, without super-admin privileges. All the descriptions I found demand that I give impersonation Rights to the domain for the service (which will never be granted by our IT)

So the question is: How do I get an access token for a single gmail inbox without user interaction (refresh token is not sufficient)?

Answer 1

UPDATE Just to avoid confusion, this is for my use case, which is very special.

• Need to integrate in a somewhat complex existing product, that already is able to communicate with multiple IMAP(S) and POP3(S) services, like Outlook365, GMX, Telekom, ... and other services (workflow system, material management, etc.). This product used googles app passwords until now to access gmail mailboxes
• Need to minimize the "new" api's, so including google specific API (in the sense of new jar files) was to be avoided, if possible
• Testing out different scenarios, but the main aim is to monitor a specific mailbox for mails with attachments by an unattended scheduler process (so no gui process)

I started to read through the available documentation and tutorials. The original aim was to migrate the Basic Authentication to OAuth2 authentication with a service account (as was done in the past for Outlook365). Unfortunately this has shown to be not feasible, due to me being unable to restrict the service account to one mailbox (I was only able to find documentation, to enable it for all mailboxes in the company; which is a no go for most IT departments). If you have no problem with opening up all mailboxes in your workspace to the app you can also use the service accounts. This has the advantage that you can avoid the first step of getting the refresh token. There is a somewhat complete explanation here: https://www.limilabs.com/blog/oauth2-gmail-imap-service-account

The following solution was for my test environment (a private, non-workspace, gmail account). Any mention of workspace features is not tested yet. It is only my collection from different tutorials and google developer pages.

The described solution consists of 2 part. One tool to create a refresh token and a second part that uses the refresh token in an unattended scheduler process. Both part use a generic library (https://www.nimbusds.com/products/nimbus-oauth-openid-connect-sdk) which is open source and already available in the product. This is no limitation, you can use any lib that provides OAuth2 flows (Authorization Code Flow and Refresh Token Flow).

To summarize my solution so far:

Go to google cloud console (https://cloud.google.com/console) and create a new application/project. In "API's and Services" you might need to activate the gmail api first.

You need to configure the "OAuth consent screen". In here you have to do a few decisions, depending on if you are in a workspace or not. In a workspace environment you can mark it as intern. This eases the deployment, since you do not need to "publish" to public. Publishing is required, if you want to deploy your application for public use.

For public use you can only use it as External/Test environment. You get the message: "Because you’re not a Google Workspace user, you can only make your app available to external (general audience) users." In Test Mode you have certain restrictions, mainly you can only use invited test accounts and all tokens are restricted to 7 days (not feasible for my use case, where the scheduler needs to run 24/7 unattended). To publish your application you need to provide (copied from google configuration screen, when you press publish):

• An official link to your app's Privacy Policy
• A YouTube video showing how you plan to use the Google user data you get from scopes
• A written explanation telling Google why you need access to sensitive and/or restricted user data
• All your domains verified in Google Search Console

So as the name suggests, Test mode is good for testing, but for deployment you need either a workspace with your own domain (internal use only) or a published app.

You need to request consent (in the "OAuth consent screen" page 2 - add or remove scopes) for using gmail api.(I used complete, since I need to move/delete mails). After that you filled out the rest of the OAuth Consent screen questions, you can create OAuth Credentials.

In "credentials" tab create a new OAuth2 client ID (I used for Desktop Application). With the collected ids and secrets you can use any lib you are familiar with to make an authcode flow. You need to set

• baseurl=https://accounts.google.com
• clientid=[your id].apps.googleusercontent.com
• clientsecret=[the client secret]
• scope=openid https://mail.google.com/

All this data is on the credentials screen, after you created it. The scope might change, if you selected e.g., only readonly access (https://www.googleapis.com/auth/gmail.readonly). With this you can execute the auth code flow and will get Access and Refresh token. Save the latest Refresh token you got. This finishes for me the "interactive" part, which a user needs to do.

After that you can use the refresh token to your (scheduler) service. The service needs to execute a refresh token flow (again, use your favorite OAuth2 lib with the settings above) to get a new access token. As mentioned above, the refresh token expires after 7 days in Test mode. Otherwise it should (not tested by me) not expire. There is a limit of refresh tokens per account (from what I read 25). If you are above this, the oldest will be deactivated. This should not be an issue, since you only need 1 token (and that should be the latest you have retrieved). The refresh token flow will not produce new refresh tokens so make sure you keep the given refresh token available (please note, other OIDC services like RedhatSSO also issue new refresh tokens with a refresh token flow, so in these cases the old one would become invalid)

The access token from the refresh token flow is (by default) 1h valid and can be used as password for imap (ssl) login. You get with the refresh token flow an expiry timestamp for the access token. After that time the access token becomes invalid. For standard java mail library it would look like:

    Properties props = new Properties();
    props.put("mail.imaps.auth.mechanisms", "XOAUTH2");
    Session session = Session.getInstance(props);
    session.setDebug(true);
    IMAPSSLStore store = new IMAPSSLStore(session, null);
    store.connect("imap.gmail.com", "your_mail@gmail.com", accessTokenString);
    Folder in = store.getFolder("Inbox");
    in.open(2);
    // do something with the messages in the folder

Please note: the given mail address must be the one you used to create the refresh token. It will not work for others.

Hope this will help someone else ;-)