High frequency data refresh with Google App Engine

Question

I'm developing a service using GAE for android clients and need to refresh application data on a regular basis, say once a minute.

In terms of architecture, this is how the application works:

• User launches app and retrieves market data from service running on GAE
• The GAE service itself calls an external web service to retrieve market data, filters results and sends for user display
• The market prices should get updated once a minute

I know GAE offers cron jobs for automated scheduled tasks, but from what I understood it's not suitable for such high frequency tasks (or not even supported)

What's the best practice/tools I can use for this use case?

Also, is it recommended to update data in the background regardless of application being open? Or just update as soon as the user launches?

[EDIT] I would also like to know if pulling data every minute is the right approach, or should the service push instead?

Thank you in advance.

Answer 1

"it's not suitable for such high frequency tasks (or not even supported)" - this is not exactly correct.

Cron jobs can run at intervals as low as 1 minute, see The schedule format:

The following are examples of schedules:

every 12 hours
every 5 minutes from 10:00 to 14:00
every day 00:00
every monday 09:00
2nd,third mon,wed,thu of march 17:00
1st monday of sep,oct,nov 17:00
1 of jan,april,july,oct 00:00

If you don't need to run a recurring job at a specific time, but instead only need to run it at regular intervals, use the form:

every N (hours|mins|minutes) ["from" (time) "to" (time)]

So for 1 min interval you can use:

every 1 minutes

If you need lower than 1 minute intervals you can use the deferred library - tasks can be delayed from the enqueueing moment with time values specified in seconds:

deferred.defer(do_something_expensive, "Foobie bletch", 12, _countdown=30, _queue="myqueue")

The answer to the last questions really depends on how you want your app to behave: have the data immediately available for the client app when the client app starts or have the client app wait until the backend collects the data.

If you're just relaying the collected data to the client either way is fine, there is no "common practice" (other than driven by higher costs for constant updates, of course). But if you plan to also offer results from processing historic data you'll probably have to go with constant updates (or maybe just during the market open hours).

Update:

The Task Queues are preferable to the deferred library, the deferred functionality is available using the optional countdown or eta arguments to taskqueue.add():

• countdown -- Time in seconds into the future that this task should run or be leased. Defaults to zero. Do not specify this argument if you specified an eta.

• eta -- A datetime.datetime that specifies the absolute earliest time at which the task should run. You cannot specify this argument if the countdown argument is specified. This argument can be time zone-aware or time zone-naive, or set to a time in the past. If the argument is set to None, the default value is now. For pull tasks, no worker can lease the task before the time indicated by the eta argument.