CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-cloud-datastore
Andrew F
Andrew F

Reputation: 595

App Engine Batch Put Too Large?

I occasionally get this error when I do batch puts.

RequestTooLargeError: The request to API call datastore_v3.Put() was too large.

The call that triggers this does a db.put call on a list of 1000+ entities. Each entity has a single db.TextProperty field, filled with about 20,000 characters. Each entity also has a parent entity, although none of the entities in the list passed to db.put share a common parent. Each of the parent entities store about 10 integers and aren't very large.

My first instinct was to split up the number of entities being passed to db.put, but

Any ideas on the cause of this?

Edit: Splitting up the entities does work. For example, I can do this:

for entity in entities: entity.put()

But the answer to this question suggests that the number of entities being put shouldn't matter. So still confused.

Upvotes: 2

Views: 749

Answers (3)

Jonny
Jonny

Reputation: 16298

I used a StringProperty until now and just ran into the same issue, when data stored in that property eventually grew too big (it just passed 1 MB). I was able to fix it for now with a JsonProperty where compressed=True. Might be an option.

Upvotes: 1

Jose Montes de Oca
Jose Montes de Oca

Reputation: 61

Is this batch update inside a transaction? if so, remember there is a 10 MB limit for a transaction size

Upvotes: 0

stevep
stevep

Reputation: 959

Remember that a quick test to find an entity's size is to try to write it to memcache. If it exceeds memcache's 1 meg limit the write will fail an you can catch the exception. May be of use if you follow Nick's suggestion to isolate the issue.

Upvotes: 4

Related Questions