When to use a certain type of persistence in Google App Engine?

Question

First of all I'll explain the question. By persistence, I mean storing data beyond the execution of a single request. It might not be the best question title, so feel free to edit it.

The way I see it, there are three types of persistence in GAE, each one "closer" to the request itself:

---

The datastore

This is where all data is most likely to be based. It may go into the higher layers of persistence temporarily, but in the end, this is where the data really is. Unfortunately, querying the datastore repeatedly is slow and uses a lot of resources.

Use when...

• storing data that should be stored for an indefinite amount of time.

Avoid using when...

• getting data that is queried often but rarely updated.

---

memcache

This is a highly complex caching engine that stores the data in memory and makes sure all users read from/write to the same cache. It's a much faster way to get/set data on a key→value basis than using the datastore. Unfortunately, data can only stay in the memory for so long, and there is no guarantee that it will stay for as long as you tell it to; the data may disappear at any time if memory is needed elsewhere.

Use when...

• you need to get data more often than you need to update it. Even when data needs to be updated often, it can have its uses (if a few missed updates are considered okay), by setting up a task queue to persist data from the memcache to the datastore.

Avoid using when...

• data needs to be updated often and has to be up-to-date when fetched.

---

Global variables

This isn't an official method of persisting data, but it works. However, it's the least reliable method, and since it has no data synchronization across servers, persisted data may show up differently for different users (but from what I've found, the server rarely changes for the same user.) Theoretically, this should be the method that has the least overhead in getting/setting values, however, and could have its uses.

Use when...

• hell freezes over? I don't know... I haven't enough knowledge about what goes on behind the scenes to actually rely on this method. Discuss!

Avoid using when...

• you rely on the data being the same across servers.

---

Cookies

If the data is user-specific, it can be efficient to store it as a cookie in the user's browser. There are some pitfalls to watch out for though:

• Security – the user can meddle with cookies, and malicious people could potentially do the same. To make sure that the contents are unreadable and unchangeable to all, the cookie can be encrypted using the PyCrypto library which is available on GAE.
• Performance – since cookies are sent with every request (even images), it can add to the bandwidth being used, and slow down requests. One solution is to use another domain for static content, so the browser won't send the cookie for that content.

---

When should the different types of persistence be used? How can they be combined to reduce/even out the amount of resources being spent?

Answer 1

Global variables

To complement AutomatedTester's answer, and also reply his further question about how to share information between GETs without memcache or datastore, below a quick illustration of how to use global variables:

if 'i' not in globals():
    i = 0

def main():
    global i
    i += 1
    print 'Status: 200'
    print 'Content-type: text/plain\n'
    print i

if __name__ == '__main__':
    main()

Calling this script multiple times will give you 1, 2, 3... Of course as mentioned earlier by Blixt you should not count on this trick too much ('i' can sometimes switch back to zero) but it can be useful to store user-specific information in a dictionary, session data for instance.

Answer 2

I use global variable to speed up json conversion. Before I convert my data structure to json, I hash it and check if the json if already available. For my app this gives quite a speedup as the pure python implementation is quite slow.

Answer 3

Your post is a good summary of the 3 major options. You mostly have answered the question already. However, if you are currently building an app and stressing over whether or not you should memcache something, try this:

1. Write your app using the datastore for everything that needs to outlive more than one request.
2. Once your app (or some usable subset) is working, run some functional tests or simulations to see where the slow spots (or high quota usage) are.
3. Find the most slow or inefficient request path, and figure out how to make that faster (either by using memcache, or altering your datastructures so you can do gets instead of queries, or possibly storing something in a global instance variable*)
4. goto 2 until you're satisfied.

*Things that might be good for a "global" variable would be something that is relatively expensive to create/fetch, that a substantial portion of your requests will use, and that does not need to be consistent across requests/users.

Answer 4

Datastore

Use the datastore to hold any long living information. The datastore should be used like you would use a normal database to hold data that will be used in your site/application.

MemCache

Use this to access data a lot quicker than trying to access the datastore. MemCache can return data really quickly and can be used for any data that needs to span multiple calls from users. It is normally data that was originally in the datastore and then moved to the memcache.

def get_data():
  data = memcache.get("key")
  if data is not None:
    return data
  else:
    data = self.query_for_data() #get data from the datastore
    memcache.add("key", data, 60)
    return data

The memcache will flush itself when the item is out of date. You set this in the last param of the add shown above.

Global Variables I wouldn't use these at all since they can't span instances. In GAE a request creates a new instance, well in python it does. If you want to use Global variables I would store the data needed in the memcache.