CODEWITHSUNDEEP
pythongoogle-app-enginememcached
Harrison
Harrison

Reputation: 830

struggling with memcache on google app engine python

I've been struggling to get memcache working on my app for a bit now. I thought I had finally got it working where it never reads from the database (unless memcache data is lost of course), only to have my site shut down because of a over-qota number of datastore reads! I'm currently using a free appspot and would like to keep it that way for as long as possible. Anyway, here's my code, maybe somebody can help me find the hole in it.

I am currently trying to implement memcache by overriding the db.Model.all(), delete(), and put() methods to query memcache first. I have memcache set up where each object in the datastore has it's own memcache value with it's id as the key. Then for each Model class I have a list of the id's under a key it knows how to query. I hope I explained this clear enough.

""" models.py """
         @classmethod
         def all(cls, order="sent"):
                 result = get_all("messages", Message)
                 if not result or memcache.get("updatemessages"):
                         result = list(super(Message, cls).all())
                         set_all("messages", result)
                         memcache.set("updatemessages", False)
                         logging.info("DB Query for messages")

                 result.sort(key=lambda x: getattr(x, order), reverse=True)
                 return result

         @classmethod
         def delete(cls, message):
                 del_from("messages", message)
                 super(Message, cls).delete(message)

         def put(self):
                 super(Message, self).put()

                 add_to_all("messages", self)



""" helpers.py """

 def get_all(type, Class):
         all = []
         ids = memcache.get(type+"allid")
         query_amount = 0
         if ids:
                 for id in ids:
                         ob = memcache.get(str(id))
                         if ob is None:
                                 ob = Class.get_by_id(int(id))
                                 if ob is None:
                                         continue
                                 memcache.set(str(id), ob)
                                 query_amount += 1
                         all.append(ob)
                 if query_amount: logging.info(str(query_amount) + " ob queries")
                 return all
         return None

 def add_to_all(type, object):
         memcache.set(str(object.key().id()), object)
         all = memcache.get(type+"allid")
         if not all:
                 all = [str(ob.key().id()) for ob in object.__class__.all()]
                 logging.info("DB query for %s" % type)
         assert all is not None, "query returned None.  Send this error code to ____: 2 3-193A"
         if not str(object.key().id()) in all:
                 all.append(str(object.key().id()))
         memcache.set(type+"allid", all)

 @log_on_fail
 def set_all(type, objects):
         assert type in ["users", "messages", "items"], "set_all was not passed a valid type.  Send this error code to ____: 33-205"
         assert not objects is None, "set_all was passed None as the list of objects.    Send this error code to _________: 33-206"
         all = []
         for ob in objects:
                 error = not memcache.set(str(ob.key().id()), ob)
                 if error:
                         logging.warning("keys not setting properly. Object must not be pickleable")
                 all.append(str(ob.key().id()))
         memcache.set(type+"allid", all)

 @log_on_fail
 def del_from(type, object):
         all = memcache.get(type+"allid")
         if not all:
                 all = object.__class__.all()
                 logging.info("DB query %s" % type)
         assert all, "Could not find any objects.  Send this error code to _____: 13-    219"
         assert str(object.key().id()) in all, "item not found in cache.  Send this error code to ________: 33-220"
         del all[ all.index(str(object.key().id())) ]
         memcache.set(type+"allid", all)
         memcache.delete(str(object.key().id()))

I apologize for all of the clutter and lack of elegance. Hopefully somebody will be able to help. I've thought about switching to ndb but for now I rather stick to my custom cache. You'll notice the logging.info("some-number of ob queries"). I get this log quite often. Maybe once or twice every half hour. Does memcache really lose data that often or is something wrong with my code?

Upvotes: 0

Views: 1611

Answers (2)

Thanos Makris
Thanos Makris

Reputation: 3115

App Engine memcache removes objects by an optimized eviction algorithm, thus having this log message with the frequency you described results in two possible explanations.

Either these data are not accessed very often or the amount of data you have in your memcache is pretty large, and thus some of it is removed from time to time.

I would also propose to move to ndb which handles the use of memcache and instance cache quite efficiently.

Hope this helps!

Upvotes: 0

bossylobster
bossylobster

Reputation: 10164

Simple solution: switch to NDB.

NDB models will store values in memcache and in an instance cache (which is 100% free) and these models will also invalidate the cache for you when you update/delete your objects. Retrieval will first try to get from the instance cache, then if that fails from memcache, and finally from the datastore, and it will set the value in any of the caches missed on the way up.

Upvotes: 5

Related Questions