Gmail python sdk : get messages by history id

Question

I have setup Google pub-sub on my gmail service account. As a result, as soon as an email is getting delivered to this email, a notification message is getting delivered to the configured pub-sub topic. Followed this guideline https://developers.google.com/gmail/api/guides/push

I've written a small Flask api to which the the notification message is sent by the Google pub-sub topic as part of subscription.

Using the message data I'm getting the history id and trying to read the message trough that history id. Below is the code snippet:

def get_messages(self, history_id):
    messages = []

    histories = self.gmail_client.users().history().list(userId=self.userId, startHistoryId=history_id).execute()

    print(histories)

    if histories and 'history' in histories:
        for history in (history for history in histories['history'] if 'messages' in history):
            for message in history['messages']:
                try:
                    message_data = self.gmail_client.users().messages().get(userId=self.userId, id=message['id']).execute()
                    messages.append(message_data)
                except Exception:
                    pass

    return messages

However, after 1-2 times initially, I'm nont gettig any messages at all with the corresponding history id.

Is there anything miissing from my side?

Answer 1

Following the documentation to receive notifications, the historyId that you are getting is the new mailbox history ID for the user, so the latest at the notification time.

If you are using this historyId from the Pub/Sub notification message to call the function get_messages(), this history ID is being passed to the users.history.list method as startHistoryId, and then as the "latest" history ID (at the time), it's very likely that there are no new delivered messages since then (unless you are getting many emails in a very short amount of time). And then as result you get no messages, you need to use a previous/older historyId, it could be from a previous notification message.

startHistoryId: string, Required. Returns history records after the specified startHistoryId. The supplied startHistoryId should be obtained from the historyId of a message, thread, or previous list response. History IDs increase chronologically but are not contiguous with random gaps in between valid IDs. Supplying an invalid or out of date startHistoryId typically returns an HTTP 404 error code. A historyId is typically valid for at least a week, but in some rare circumstances may be valid for only a few hours. If you receive an HTTP 404 error response, your application should perform a full sync. If you receive no nextPageToken in the response, there are no updates to retrieve and you can store the returned historyId for a future request.

Something to keep in mind here, is that your code seems fine; however, without the input used to call the function get_messages(), and the output from the method users.history.list, it's a bit difficult to know what's happening when you run your code. I would suggest to keep track of the previous historyId and use them, instead of using immediately the historyId that you are getting from the Pub/Sub notification message.

When sharing the input & output, please make sure to remove any sensitive or personal data that could be included.

Some additional pointers on your code, according to the documentation, the users.history.list method returns an object of the form:

{
  "historyId": "A String",
  "nextPageToken": "A String",
  "history": [
    {
      "labelsRemoved": [
        {
          "message": { # An email message.
          },
          "labelIds": [
            "A String",
          ],
        },
      ],
      "messagesAdded": [
        {
          "message": { # An email message.
          },
        },
      ],
      "labelsAdded": [
        {
          "message": { # An email message.
          },
          "labelIds": [
            "A String",
          ],
        },
      ],
      "id": "A String",
      "messagesDeleted": [
        {
          "message": { # An email message.
          },
        },
      ],
      "messages": [
        {  # An email message.
        },
      ],
    },
  ],
}

Where the message (email message) objects have the following structure:

"message": { # An email message.
  "threadId": "A String",
  "labelIds": [
    "A String",
  ],
  "sizeEstimate": 42,
  "id": "A String",
  "raw": "A String",
  "payload": {
    "mimeType": "A String",
    "partId": "A String",
    "parts": [],
    "headers": [
      {
        "name": "A String",
        "value": "A String",
      },
    ],
    "body": {
      "data": "A String",
      "size": 42,
      "attachmentId": "A String",
    },
    "filename": "A String",
  },
  "snippet": "A String",
  "historyId": "A String",
  "internalDate": "A String",
}

On the other hand the users.messages.get method returns an email message object with the same structure (see previous snippet).

Based on this, if you are getting the message IDs from the users.history.list method (e.g. history['messages'][0]['id']), then you are also getting the email message objects. Thus, what you are trying to do by iterating the list history['messages'] to get every message by ID using the users.messages.get method, is the same as doing the following:

messages = history['messages']

Or if you would like to get something more specific as the content of body.data under payload you could use:

history['messages'][0]['payload']['body']['data']

# The index [0] is to point to the first message on the list, 
# if you are iterating the list, the index is not necessary
# e.g. message['payload']['body']['data']

This way, you don't need to iterate over the message IDs and use the users.messages.get method to get one by one the email messages, as you already have them from the users.history.list method.