CODEWITHSUNDEEP
javagoogle-app-enginestoragetemporary-objects
nanofarad
nanofarad

Reputation: 41281

Storing a list of objects on appengine across all instances

I have an app on Google App Engine. It processes realtime data without any regard for past data(so no datastore/full persistency), and thus I don't need anything extremely reliable to recover data in case of an instance crash or new deployment.

I have multiple instances, due to the load, that need to share a single list of objects. I looked into memcached, but this requires a mapping from key to object in a map-style interface without the ability to dump all entries.

In addition, I'm hoping to make the entries expirable unless "bumped", though that can handle in user code.

What's a reasonable solution to this that doesn't involve the datastore? I'd prefer for it to be able to handle concurrent reads and writes, from multiple app engine instances(which is why a simple Java object declaration doesn't work as it's per-instance).

Memcached could potentially store a single object but it would require getting the object, getting some sort of lock, modifying it, and writing it back to the cache.

N.B. There is unrelated data in the presistent app engine datastore.

Clarification: I need all the entities. Entities themselves may be modified at any time, removed by expiration or semi-automatically, or created from scratch. I'm not particularly able to head for billable backends.

Edit: Yes, it would work with single lists stored locally but I expect spikes where multiple instances are needed. I'd need at least a best-effort approach to getting a full list, though I don't mind a delay of synchronization. However, round-robining communication between all instances sans backend would be relatively unscalable, and I'd be stuck in more of a rent-a-server scenario there.

I don't care if changes don't make it through immediately after a change is committed on another frontend instance. The changes would need to make it through with 100% certainty, however. I only see insertions and updates where a lastSeen value(long) is increased. The updates should also be replicated or reflected on all instances.

Note: The bounty got refunded due to the fact that I will unable to award it due to circumstances at that time, that I cannot control. I will re-add when possible.

Upvotes: 0

Views: 290

Answers (1)

rattray
rattray

Reputation: 6069

What's wrong with the datastore here?

If your concern is that you'll rack up too much expense having a lot of items in the db, just save each item with a datetime_created field with auto_now_add=True (or the Java equivalent; I come from Python), and schedule a cron to go through and delete anything old?

The Datastore is supposed to be highly available for concurrent reads and writes, as far as I'm aware.

You might also consider just moving off AppEngine. Redis sounds like the right tool for the job you're describing, and Elastic Beanstalk on AWS could probably handle your needs quite well there. Of course, it'd be more effort to get things set up, but likely not overwhelmingly so.

Upvotes: 0

Related Questions