Office 365 IMAP authentication via OAuth2 and python MSAL library

Question

I'm trying to upgrade a legacy mail bot to authenticate via Oauth2 instead of Basic authentication, as it's now deprecated two days from now.

The document states applications can retain their original logic, while swapping out only the authentication bit

Application developers who have built apps that send, read, or otherwise process email using these protocols will be able to keep the same protocol, but need to implement secure, Modern authentication experiences for their users. This functionality is built on top of Microsoft Identity platform v2.0 and supports access to Microsoft 365 email accounts.

Note I've explicitly chosen the client credentials flow, because the documentation states

This type of grant is commonly used for server-to-server interactions that must run in the background, without immediate interaction with a user.

So I've got a python script that retrieves an Access Token using the MSAL python library. Now I'm trying to authenticate with the IMAP server, using that Access Token. There's some existing threads out there showing how to connect to Google, I imagine my case is pretty close to this one, except I'm connecting to a Office 365 IMAP server. Here's my script

import imaplib
import msal
import logging

app = msal.ConfidentialClientApplication(
    'client-id',
    authority='https://login.microsoftonline.com/tenant-id',
    client_credential='secret-key'
)

result = app.acquire_token_for_client(scopes=['https://graph.microsoft.com/.default'])

def generate_auth_string(user, token):
  return 'user=%s\1auth=Bearer %s\1\1' % (user, token)

# IMAP time!
mailserver = 'outlook.office365.com'
imapport = 993
M = imaplib.IMAP4_SSL(mailserver,imapport)
M.debug = 4
M.authenticate('XOAUTH2', lambda x: generate_auth_string('user@mydomain.com', result['access_token']))

print(result)

The IMAP authentication is failing and despite setting M.debug = 4, the output isn't very helpful

  22:56.53 > b'DBDH1 AUTHENTICATE XOAUTH2'
  22:56.53 < b'+ '
  22:56.53 write literal size 2048
  22:57.84 < b'DBDH1 NO AUTHENTICATE failed.'
  22:57.84 NO response: b'AUTHENTICATE failed.'
Traceback (most recent call last):
  File "/home/ubuntu/mini-oauth.py", line 21, in <module>
    M.authenticate("XOAUTH2", lambda x: generate_auth_string('user@mydomain.com', result['access_token']))
  File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
    raise self.error(dat[-1].decode('utf-8', 'replace'))
imaplib.IMAP4.error: AUTHENTICATE failed.

Any idea where I might be going wrong, or how to get more robust information from the IMAP server about why the authentication is failing?

Things I've looked at

• Note this answer no longer works as the suggested scopes fail to generate an Access Token.

• The client credentials flow seems to mandate the https://graph.microsoft.com/.default grant. I'm not sure if that includes the scope required for the IMAP resource https://outlook.office.com/IMAP.AccessAsUser.All?

• Verified the code lifted from the Google thread produces the SASL XOAUTH2 string correctly, per example on the MS docs

import base64

user = 'test@contoso.onmicrosoft.com'
token = 'EwBAAl3BAAUFFpUAo7J3Ve0bjLBWZWCclRC3EoAA'

xoauth = "user=%s\1auth=Bearer %s\1\1" % (user, token)

xoauth = xoauth.encode('ascii')
xoauth = base64.b64encode(xoauth)
xoauth = xoauth.decode('ascii')

xsanity = 'dXNlcj10ZXN0QGNvbnRvc28ub25taWNyb3NvZnQuY29tAWF1dGg9QmVhcmVyIEV3QkFBbDNCQUFVRkZwVUFvN0ozVmUwYmpMQldaV0NjbFJDM0VvQUEBAQ=='

print(xoauth == xsanity) # prints True

• This thread seems to suggest multiple tokens need to be fetched, one for graph, then another for the IMAP connection; could that be what I'm missing?

Answer 1

After a wee struggle (non-microsoft user) I managed to get sardar-agabejli example code to authenticate. My problem was not understanding what setting up the Service Principal meant. From ubuntu linux I needed to:

$ snap install powershell
$ pwsh
ps> install-module exchangeonlinemanagement
ps> Connect-ExchangeOnline
ps> New-ServicePrincipal -AppId <appid> -ObjectId <objid>
ps> Add-MailboxPermission -Identity <email@domain> -User <ObjectId> -AccessRights FullAccess`
ps> exit
$

Look up application id and object id in azure as described above. If you fail to enter the final powershell command you will authenticate but not connect causing this error: SELECT command error: BAD [b'User is authenticated but not connected.']

Answer 2

I wasn't able to get any of the above solutions to work. It seems to me that Microsoft doesn't really want you to interact with your office365 email account via IMAP anymore and instead wants you to use the Microsoft Graph Outlook REST API instead. The steps to set things up this way are simpler and I personally find the API easier to interact with than IMAP.

1. add the Microsoft Graph Mail.ReadWrite permission as shown here
2. use @amit's code to get_access_token() and the interact with your mail using requests... change the scope to be 'https://graph.microsoft.com/.default'
3. Interact with mail via the Graph API

import requests

access_token = get_access_token()  # from @amit's answer above
base_url = "https://graph.microsoft.com/v1.0"

# example url to list folders for a user's mailbox
url = f"{base_url}/users/{user_id}/mailFolders"
response = requests.get(
     url, 
     headers={
          'Authorization': 'Bearer ' + access_token['access_token']
     }
)

Answer 3

Try the below steps.

For Client Credentials Flow you need to assign “Application permissions” in the app registration, instead of “Delegated permissions”.

1. Add permission “Office 365 Exchange Online / IMAP.AccessAsApp” (application).
2. Grant admin consent to you application
3. Service Principals and Exchange.
4. Once a service principal is registered with Exchange Online, administrators can run the Add-Mailbox Permission cmdlet to assign receive permissions to the service principal.
5. Use scope 'https://outlook.office365.com/.default'.

Now you can generate the SALS authentication string by combining this access token and the mailbox username to authenticate with IMAP4.

#Python code

def get_access_token():
    tenantID = 'abc'
    authority = 'https://login.microsoftonline.com/' + tenantID
    clientID = 'abc'
    clientSecret = 'abc'
    scope = ['https://outlook.office365.com/.default']
    app = ConfidentialClientApplication(clientID, 
          authority=authority, 
          client_credential = clientSecret)
    access_token = app.acquire_token_for_client(scopes=scope)
    return access_token

def generate_auth_string(user, token):
    auth_string = f"user={user}\1auth=Bearer {token}\1\1"
    return auth_string

#IMAP AUTHENTICATE
 imap = imaplib.IMAP4_SSL(imap_host, 993)
 imap.debug = 4
 access_token = get_access_token_to_authenticate_imap()
 imap.authenticate("XOAUTH2", lambda x:generate_auth_string(
      'useremail',
       access_token['access_token']))
 imap.select('inbox')

Answer 4

The imaplib.IMAP4.error: AUTHENTICATE failed Error occured because one point in the documentation is not that clear.

When setting up the the Service Principal via Powershell you need to enter the App-ID and an Object-ID. Many people will think, it is the Object-ID you see on the overview page of the registered App, but its not! At this point you need the Object-ID from "Azure Active Directory -> Enterprise Applications --> Your-App --> Object-ID"

New-ServicePrincipal -AppId <APPLICATION_ID> -ServiceId <OBJECT_ID> [-Organization <ORGANIZATION_ID>]

Microsoft says:

The OBJECT_ID is the Object ID from the Overview page of the Enterprise Application node (Azure Portal) for the application registration. It is not the Object ID from the Overview of the App Registrations node. Using the incorrect Object ID will cause an authentication failure.

Ofcourse you need to take care for the API-permissions and the other stuff, but this was for me the point. So lets go trough it again, like it is explained on the documentation page. Authenticate an IMAP, POP or SMTP connection using OAuth

1. Register the Application in your Tenant
2. Setup a Client-Key for the application
3. Setup the API permissions, select the APIs my organization uses tab and search for "Office 365 Exchange Online" -> Application permissions -> Choose IMAP and IMAP.AccessAsApp
4. Setup the Service Principal and full access for your Application on the mailbox
5. Check if IMAP is activated for the mailbox

Thats the code I use to test it:

import imaplib
import msal
import pprint

conf = {
    "authority": "https://login.microsoftonline.com/XXXXyourtenantIDXXXXX",
    "client_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", #AppID
    "scope": ['https://outlook.office365.com/.default'],
    "secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-Value
    "secret-id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-ID
}
    
def generate_auth_string(user, token):
    return f"user={user}\x01auth=Bearer {token}\x01\x01"    

if __name__ == "__main__":
    app = msal.ConfidentialClientApplication(conf['client_id'], authority=conf['authority'],
                                             client_credential=conf['secret'])

    result = app.acquire_token_silent(conf['scope'], account=None)

    if not result:
        print("No suitable token in cache.  Get new one.")
        result = app.acquire_token_for_client(scopes=conf['scope'])

    if "access_token" in result:
        print(result['token_type'])
        pprint.pprint(result)
    else:
        print(result.get("error"))
        print(result.get("error_description"))
        print(result.get("correlation_id"))
        
    imap = imaplib.IMAP4('outlook.office365.com')
    imap.starttls()
    imap.authenticate("XOAUTH2", lambda x: generate_auth_string("target_mailbox@example.com", result['access_token']).encode("utf-8"))

After setting up the Service Principal and giving the App full access on the mailbox, wait 15 - 30 minutes for the changes to take effect and test it.

Answer 5

Try with this script:

import json
import msal

import requests

client_id = '***'
client_secret = '***'
tenant_id = '***'
authority = f"https://login.microsoftonline.com/{tenant_id}"

app = msal.ConfidentialClientApplication(
    client_id=client_id,
    client_credential=client_secret,
    authority=authority)

scopes = ["https://graph.microsoft.com/.default"]

result = None
result = app.acquire_token_silent(scopes, account=None)

if not result:
    print(
        "No suitable token exists in cache. Let's get a new one from Azure Active Directory.")
    result = app.acquire_token_for_client(scopes=scopes)

# if "access_token" in result:
#     print("Access token is " + result["access_token"])


if "access_token" in result:
    userId = "***"
    endpoint = f'https://graph.microsoft.com/v1.0/users/{userId}/sendMail'
    toUserEmail = "***"
    email_msg = {'Message': {'Subject': "Test Sending Email from Python",
                             'Body': {'ContentType': 'Text', 'Content': "This is a test email."},
                             'ToRecipients': [{'EmailAddress': {'Address': toUserEmail}}]
                             },
                 'SaveToSentItems': 'true'}
    r = requests.post(endpoint,
                      headers={'Authorization': 'Bearer ' + result['access_token']}, json=email_msg)
    if r.ok:
        print('Sent email successfully')
    else:
        print(r.json())
else:
    print(result.get("error"))
    print(result.get("error_description"))
    print(result.get("correlation_id"))

Source: https://kontext.tech/article/795/python-send-email-via-microsoft-graph-api