CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
Verilocos
Verilocos

Reputation: 115

Google App Engine - What's the recommended way to keep entity amount within limit?

I have some entities of a kind, and I need to keep them within limited amount, by discarding old ones. Just like log entries maintainance. Any good approach on GAE to do this?

Options in my mind:

Option 1. Add a Date property for each of these entities. Create cron job to check datastore statistics daily. If it exceeds the limit, query some entities of that kind and sort by date with oldest first. Delete them until the size is less than, for example, 0.9 * max_limit.

Option 2. Option 1 requires an additional property with index. I observed that the entity key ids may be likely increasing. So I'd like to query only keys and sort by ascending order. Delete the ones with smaller ids. It does not require additional property (date) and index. But I'm seriously worrying about whether the key id is assured to go increasingly?

I think this is a common data maintainance task. Is there any mature way to do it?

By the way, a tiny ad for my app, free and purely for coder's fun! http://robotypo.appspot.com

Upvotes: 1

Views: 572

Answers (2)

maddyblue
maddyblue

Reputation: 16882

You cannot assume that the IDs are always increasing. The docs about ID gen only guarantee that:

IDs allocated in this manner will not be used by the Datastore's automatic ID sequence generator and can be used in entity keys without conflict.

The default sort order is also not guaranteed to be sorted by ID number:

If no sort orders are specified, the results are returned in the order they are retrieved from the Datastore.

which is vague and doesn't say that the default order is by ID.

One solution may be to use a rotating counter that keeps track of the first element. When you want to add new entities: fetch the counter, increment it, mod it by the limit, and add a new element with an ID as the value of the counter. This must all be done in a transaction to guarantee that the counter isn't being incremented by another request. The new element with the same key will overwrite one that was there, if any.

When you want to fetch them all, you can manually generate the keys (since they are all known), do a bulk fetch, sort by ID, then split them into two parts at the value of the counter and swap those.

If you want the IDs to be unique, you could maintain a separate counter (using transactions to modify and read it so you're safe) and create entities with IDs as its value, then delete IDs when the limit is reached.

Upvotes: 1

Thanos Makris
Thanos Makris

Reputation: 3115

You can create a second entity (let's call it A) that keeps a list of the keys of the entities you want to limit, like this (pseudo-code):

class A:
    List<Key> limitedEntities;

When you add a new entity, you add its key in the list of A. If the length of the list exceeds the limit, you take the first element of the list and the remove the corresponding entity.

Notice that when you add or delete an entity, you should modify the list of entity A in a transaction. Since, these entities belong to different entity groups, you should consider using Cross-Group Transactions.

Hope this helps!

Upvotes: 1

Related Questions