580
580

Reputation: 480

Google.Apis.Admin.Email_Migration_v2 [HTTP Status Code 412 – Limit Reached]

Edit 2:

Client Library: After reviewing it is not easily suggested that this is for the .NET client library.

DLL: Google.Apis.Admin.email_migration_v2.dll


What steps will reproduce the problem?

  1. Generate a process which contains a Google.Apis.Admin.email_migration_v2.AdminService instance for each unique Google Apps Gmail mailbox that will have messages sent to it. All of the AdminService objects generated use the same OAuth2.0 credentials and application name. Each AdminService object generated will only send messages to one Google Apps user’s mailbox. For example, if we were sending messages to five different Google Apps Gmail mailboxes we would generate five AdminService objects to send messages; one for each user’s mailbox.

    • Biggest thing to note is that each AdminService object created is created on a separate process.

    • AdminService objects were given a FileDataStore object to change the location of where the refresh token is stored; C:\ProgramData\SomeFile\SomeFile.

    • Supplied appropriate scopes to the credentials.

  2. Begin sending mail messages on each process. Using one thread to send messages in each process, so only one message is sent at a time to each user’s mailbox.

    • Each message sent gets its own instance of MailItem and MailResource.InsetMedia

    • The MailResource.InsertMedia object is generated for each item by calling AdminService.Mail.Insert(MailItem, string, Stream, string) method.

  3. When our code makes the call to MailResource.InsertMediaUpload.UploadAsync(CancellationTokenSource).Result is where we can receive the error.

    • The error is caught and handled (logged) from the return type of the aforementioned call; the type is Google.Apis.Upload.IUploadProgress. The exception is handled using the IUploadProgress.Exception property.

What is the expected output? What do you see instead?

What version of the product are you using?

What is your operating system?

What is your IDE?

What is the .NET framework version?

Please provide any additional information below.

Questions:

  1. So what exactly does this 412 http status code mean? What limit is being encountered that this message is referring to?
  2. Shouldn’t we be receiving some form of 5XX error from the server if we are hitting a limit? In which case wouldn’t the built in exponential back off policy kick in?
    • a. Unless the server is checking the POST request for a pre-condition about a server side limit then telling the client to back off which is what a 412 error seems to typically indicate. In that case please give as much detail as possible for question 1.

Sorry for the extensive post! Thanks for your time! I will also be creating a defect/issue in Google's .NET issue tracker and providing a link.


Edit 1:

For anyone interested in following this issue here is a link to the submitted item in Google's issue tracker for .NET. Submitted Issue

For reference it is issue 492.

Upvotes: 2

Views: 650

Answers (2)

580
580

Reputation: 480

I believe I have found an answer to this problem, though I will advise a disclaimer, I do not work for Google and cannot be 100% sure of the accuracy; you've been warned. This should at least hold true for the .NET version of Google's Email Migration v2 API. I cannot guarantee how other APIs work because I do not use them..

Through working with this API in spurts for well over eight months now, it appears that if an application or multiple applications are to send messages to a single Google Apps user/mailbox consistently, at a faster rate than which Google servers can process, then at some rate you should start to get a bunch of GoogleApiExceptions stating "412 - Limit Reached" when sending new messages. What we have gathered through using our application is that each Google Apps user/mailbox has its own pending items queue. When you send a message to Google Apps it is first put into this queue before being processed by a Google Server and put into the user's mailbox. If this queue becomes full and you attempt to send another message you will receive a 412 error.

Options are to wait before sending another message, you'll have to wait however long the Google server takes to process the next message in the user's queue before sending another; which is unpredictable. The better option in my opinion is to start sending messages to another Google Apps user; because each user appears to have its own message queue. Be sure to stop sending to the user who is consistently getting 412 errors. This will give the Google server some time to process that user's packed message queue. Note each pending messages queue appeared to hold about 100-150 items before throwing 412 errors.

503 errors appear to occur when sending messages into a user's mailbox queue at a higher rate than 1 request per second. As Emily has stated "the QPS limit you see in the Google Developers Console project is not the actual default limit" it is truly 1 QPS per Google Apps user.

As for the exponential back-off it is supposed to be implemented automatically see this. Note Peleyal appears to be the gentleman in charge of the API; can be noted from the download page for the API.

This took us a little while to figure out so cheers if you're having this issue! Please if you find any contradicting information correct any mistakes found in this answer or make your own!!

Upvotes: 0

Emily
Emily

Reputation: 1474

I am not quite sure where you see the "the specified limit of 20 requests a second for the Email Migration API". Reminder: the QPS limit you see in the Google Developers Console project is not the actual default limit. You can change that limit to anything you want, and thus, that's not the actual limit for the API. It is really just for managing the consumption of the API quota (some APis will have a much higher QPS where you can adjust it to lower for different projects across your console).

According to the email migration APi documentation, the QPS is 1 request per second (the link is here: https://developers.google.com/admin-sdk/email-migration/v2/limits).

I have experienced 412 errors when the QPS limit is being hit, and I have also seen the 412 error returned when I am uploading too much data to a single domain. How much data are you loading all at once? I would suggest doing an exponential backoff to see if the issue would disappear.

Upvotes: 0

Related Questions