Consolidating Results of Tasks running on GAE (Java) Push Queue

Question

I want to put tasks onto a push queue in my GAE java app to audit the status of my users, 1 user per task. I am struggling to find a good solution for how to keep track of and consolidate the results of the audit in a safe, efficient way, across each of the tasks.

Currently what I do is use a deferred task that has one parameter in its constructor: the string representation of the key of the entity representing the user I am trying to audit.

The types of data I need to hold for each user are yes/no things like are they active, are their connections to other services revoked/not revoked, etc. When I consolidate, all I want to do is add up all the yes's and no's for each question, so e.g. I have 1000 active users, and 200 inactive users, etc.

Approaches I have considered (please excuse naivety...) are:

• Each task puts its own entity, which I then check afterwards to consolidate the results of all the tasks
  • Pros: thread safe and safe in a distributed environment
  • Cons: relatively costly (i.e. because I have to write a lot of rows, and then later delete them), and a nuisance because I then have to post-process the entity to consolidate the data from each of the rows
• I could grab an instance of some singleton class within each deferred task, and use that to hold and consolidate the data, and then cause it to persist its data later
  • Pros: cheap (don't have to write/delete extra data), tidy because the single object does all the pieces of work, can be made thread safe
  • Cons: on GAE I can't (easily?) rely on the singleton really being a singleton because it's distributed, so in practice I could end up with data being consolidated in two or more similar "singleton" objects
• I suspect I could write the data into some predefined context object (the servlet context?), or memcache?
  • Pros: cheap, fast, potentially simple?
  • Cons: memcache data could get overridden - it's not safe; not sure about servlet context

Bottom line is all of these are my made up approaches, and I am sure there must be a standard approach / pattern for doing this type of thing... but I haven't been able to find it.

Any advice please...? Thanks!

Answer 1

There are a few common approaches:

DataStore counters https://developers.google.com/appengine/articles/sharding_counters

MemCache counters https://developers.google.com/appengine/articles/scaling/memcache#transient http://blog.notdot.net/2010/04/High-concurrency-counters-without-sharding

Pull queue and task tags http://www.youtube.com/watch?v=AM0ZPO7-lcE