CODEWITHSUNDEEP
javagoogle-app-enginegoogle-cloud-datastore
Jordi P.S.
Jordi P.S.

Reputation: 4018

Google App Engine java OutOfMemoryError when putAll on memcache

I'm storing a map of around 20k entries on GAE memchace. Each entry it's around 1Kb big. I'm getting the following error. Is there a limitation? In my understanding the limitation of 1Mb it's for every entity you put in the memcache not the whole batch.

java.lang.OutOfMemoryError: Java heap space
at com.google.appengine.repackaged.com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:34)
at com.google.appengine.api.memcache.MemcacheServiceApiHelper.makeAsyncCall(MemcacheServiceApiHelper.java:104)
at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl.doPutAll(AsyncMemcacheServiceImpl.java:521)
at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl.putAll(AsyncMemcacheServiceImpl.java:564)
at com.google.appengine.api.memcache.MemcacheServiceImpl.putAll(MemcacheServiceImpl.java:112)
[...]

And my code looks like:

final HashMap<EntityToStoreKey, EntityToStore> map = new HashMap<EntityToStoreKey, EntityToStore>();
for (EntityToStore entityToStore : entitiesToStore) {
    index.add(entityToStore);
    map.put(EntityToStoreKey.key(entityToStore.getId(), false), entityToStore);
}
entityToStoreCache.putAll(map, portalCacheTimeout);

And the issue only happens on a prod environment, not in local.

Any help/tip ?

Upvotes: 1

Views: 972

Answers (3)

ZiglioUK
ZiglioUK

Reputation: 2610

Are you using Appstats? I've found Appstats uses a large chunk of memory, see my question here: How to reduce the memory usage of Appstats on Google App Engine Java

I've experienced OutOfMemoryError when trying to load around 20,000 entries at once, similar size as yours. Now I load them in batches, 1000 at the time. Here's an example code using Siena (but it should be easy to port to other ORMs): http://groups.google.com/group/siena-discuss/msg/cd874589ef7aaa66

I find handy to use the method List.subList() followed by List.remove() or List.clear()

[Update]

I've found an undocumented parameter that limits the size of StackTrace:

<filter>
  <filter-name>appstats</filter-name>
    <filter-class>com.google.appengine.tools.appstats.AppstatsFilter</filter-class>
    <init-param>
      <param-name>maxLinesOfStackTrace</param-name>
      <param-value>16</param-value>
    </init-param>
</filter>

Upvotes: 1

Nick Johnson
Nick Johnson

Reputation: 101149

You're getting an exception because your Java servlet container is running out of memory, not because of anything related to memcache.

If you're trying to set 20,000 memcache keys in a single request, though, I suspect you're Doing It Wrong.

Upvotes: 1

Rick Mangi
Rick Mangi

Reputation: 3769

The size of memcache is a total black box in GAE. Don't trust it to be anything. You are correct, no single entity can be larger than 1 MB but you have NO IDEA how big your total memcache is nor do you have any idea what the eviction policy is.

The latest release of GAE seems to be giving us some visibility into memcache but up until now it's really been a black box. I recommend not doing what you're trying to do. If you need to warm your cache do it in a loop and only load the most relevant content.

Upvotes: 2

Related Questions