How to keep global variables persistent over multiple google appengine instances?

Question

Our situation is as follows: We are working on a schoolproject where the intention is that multiple teams walk around in a city with smarthphones and play a city game while walking. As such, we can have 10 active smarthpones walking around in the city, all posting their location, and requesting data from the google appengine.

Someone is behind a webbrowser,watching all these teams walk around, and sending them messages etc.

We are using the datastore the google appengine provides to store all the data these teams send and request, to store the messages and retrieve them etc. However we soon found out we where at our max limit of reads and writes, so we searched for a solution to be able to retrieve periodic updates(which cost the most reads and writes) without using any of the limited resources google provides. And obviously, because it's a schoolproject we don't want to pay for more reads and writes.

Storing this information in global variables seemed an easy and quick solution, which it was... but when we started to truly test we noticed some of our data was missing and then reappearing. Which turned out to be because there where so many requests being done to the cloud that a new instance was made, and instances don't keep these global variables persistent.

So our question is: Can we somehow make sure these global variables are always the same on every running instance of google appengine. OR Can we limit the amount of instances ever running, no matter how many requests are done to '1'. OR Is there perhaps another way to store this data in a better way, without using the datastore and without using globals.

Answer 1

First, definitely use memcache as Tim Delaney said. That alone will probably solve your problem.

If not, you should consider a push model. The advantage is that your clients won't be asking you for new data all the time, only when something has actually changed. If the update is small enough that you can deliver it in the push message, you won't need to worry about datastore reads on memcache misses, or any other duplicate work, for all those clients: you read the data once when it changes and push it out to everyone.

The first option for push is C2DM (Android) or APN (iOS). These are limited on the amount of data they send and the frequency of updates.

If you want to get fancier you could use XMPP instead. This would let you do more frequent updates with (I believe) bigger payloads but might require more engineering. For a starting point, see Stack Overflow questions about Android and iOS.

Have fun!

Answer 2

You should be using memcache. If you use the ndb (new database) library, you can automatically cache the results of queries. Obviously this won't improve your writes much, but it should significantly improve the numbers of reads you can do.

You need to back it with the datastore as data can be ejected from memcache at any time. If you're willing to take the (small) chance of losing updates you could just use memcache. You could do something like store just a message ID in the datastore and have the controller periodically verify that every message ID has a corresponding entry in memcache. If one is missing the controller would need to reenter it.

Answer 3

Consider jabber protocol for communication between peers. Free limits are on quite high level for it.

Answer 4

Interesting question. Some bad news first, I don't think there's a better way of storing data; no, you won't be able to stop new instances from spawning and no, you cannot make seperate instances always have the same data.

What you could do is have the instances perioidically sync themselves with a master record in the datastore, by choosing the frequency of this intelligently and downloading/uploading the information in one lump you could limit the number of read/writes to a level that works for you. This is firmly in the kludge territory though.

Despite finding the quota for just about everything else, I can't find the limits for free read/write so it is possible that they're ludicrously small but the fact that you're hitting them with a mere 10 smartphones raises a red flag to me. Are you certain that the smartphones are being polled (or calling in) at a sensible frequency? It sounds like you might be hammering them unnecessarily.