Reputation: 1157
I am getting the following error from a batch request to the GMail API on Google App Engine:
RequestTooLargeError: The request to API call datastore_v3.Put() was too large.
From other questions on Stackoverflow I understand that the problem has to do with memcache. Nevertheless I don't know how to solve this issue, since it is coming up even if I run 1 request per batch and it is before I can actually do something with the content of the email (like compressing it).
My code currently looks as follows:
count = 0 #start a new batch request after every 1000 requests
batch = BatchHttpRequest(callback=get_items)
for i in new_items:
batch.add(service.users().messages().get(userId=email, id=i), request_id=str(count))
count += 1
if count % 1000 == 0:
for n in range(0, 5):
try:
batch.execute(http=http)
break
except Exception as e:
if n < 4:
time.sleep((2 ** n) + random.randint(0, 1000) / 1000)
else:
raise
batch = BatchHttpRequest(callback=get_items)
if count % 1000 != 0:
for n in range(0, 5):
try:
batch.execute(http=http)
break
except Exception as e:
if n < 4:
time.sleep((2 ** n) + random.randint(0, 1000) / 1000)
else:
raise
What would be a possible solution?
EDIT
Adding callback function
def get_items(request_id, response, exception):
if exception is not None:
print 'An error occurred: %s' % exception
else:
save_messages = request_id.split('/', 2)[1]
email = request_id.split('/', 2)[2]
in_reply_to = ''
m_id = ''
for r in response['payload']['headers']:
if r['name'].lower() == 'message-id':
m_id = r.get('value')
elif r['name'].lower() == 'in-reply-to':
in_reply_to = r.get('value')
elif r['name'].lower() == 'from':
sender, t_t = stripEmails(r.get('value'), None, None, True, email, False, False)
if m_id:
incoming = Gmail(id=m_id) #skip if exists already
else:
logging.info(response)
logging.exception('No message ID detect')
return
incoming.email = email
incoming.response = json.dumps(response)
incoming.put()
Upvotes: 1
Views: 137
Reputation: 1157
By savings the too large text item as JsonProperty with compressed=true the problem was solved.
Upvotes: 1