Vincent
Vincent

Reputation: 1157

Gmail API Python: RequestTooLargeError: The request to API call datastore_v3.Put() was too large

I am getting the following error from a batch request to the GMail API on Google App Engine:

RequestTooLargeError: The request to API call datastore_v3.Put() was too large.

From other questions on Stackoverflow I understand that the problem has to do with memcache. Nevertheless I don't know how to solve this issue, since it is coming up even if I run 1 request per batch and it is before I can actually do something with the content of the email (like compressing it).

My code currently looks as follows:

          count = 0 #start a new batch request after every 1000 requests
          batch = BatchHttpRequest(callback=get_items)
          for i in new_items:
            batch.add(service.users().messages().get(userId=email, id=i), request_id=str(count))
            count += 1
            if count % 1000 == 0:
              for n in range(0, 5): 
                try:
                  batch.execute(http=http)
                  break
                except Exception as e:
                  if n < 4:
                    time.sleep((2 ** n) + random.randint(0, 1000) / 1000)
                  else:
                    raise
              batch = BatchHttpRequest(callback=get_items) 
          if count % 1000 != 0:
            for n in range(0, 5): 
              try:
                batch.execute(http=http)
                break
              except Exception as e:
                if n < 4:
                  time.sleep((2 ** n) + random.randint(0, 1000) / 1000)
                else:
                  raise

What would be a possible solution?

EDIT

Adding callback function

def get_items(request_id, response, exception):
  if exception is not None:
      print 'An error occurred: %s' % exception
  else:
      save_messages = request_id.split('/', 2)[1]
      email = request_id.split('/', 2)[2]
      in_reply_to = ''
      m_id = ''
      for r in response['payload']['headers']:
          if r['name'].lower() == 'message-id':
              m_id = r.get('value')
          elif r['name'].lower() == 'in-reply-to':
              in_reply_to = r.get('value')
          elif r['name'].lower() == 'from':
              sender, t_t = stripEmails(r.get('value'), None, None, True, email, False, False)
      if m_id:
        incoming = Gmail(id=m_id) #skip if exists already
      else:
        logging.info(response)
        logging.exception('No message ID detect')
        return
      incoming.email = email
      incoming.response = json.dumps(response)
      incoming.put()

Upvotes: 1

Views: 137

Answers (1)

Vincent
Vincent

Reputation: 1157

By savings the too large text item as JsonProperty with compressed=true the problem was solved.

Upvotes: 1

Related Questions