CODEWITHSUNDEEP
pythonmultithreadinggoogle-app-enginequeue
RobertPitt
RobertPitt

Reputation: 57268

Google AppEngine and Threaded Workers

I am currently trying to develop something using Google AppEngine, I am using Python as my runtime and require some advise on setting up the following.

I am running a webserver that provides JSON data to clients, The data comes from an external service in which I have to pull the data from.

What I need to be able to do is run a background system that will check the memcache to see if there are any required ID's, if there is an ID I need to fetch some data for that ID from the external source and place the data in the memecache.

If there are multiple id's, > 30 I need to be able to pull all 30 request as quickly and efficiently as possible.

I am new to Python Development and AppEngine so any advise you guys could give would be great.

Thanks.

Upvotes: 0

Views: 156

Answers (2)

Guido van Rossum
Guido van Rossum

Reputation: 16890

Note that using memcache as the communication mechanism between front-end and back-end is unreliable -- the contents of memcache may be partially or fully erased at any time (and it does happen from time to time).

Also note that you can't query memcache of you don't know the exact keys ahead of time. It's probably better to use the task queue to queue up requests instead of using memcache, or using the datastore as a storage mechanism.

Upvotes: 1

Riley Lark
Riley Lark

Reputation: 20890

You can use "backends" or "task queues" to run processes in the background. Tasks have a 10-minute run time limit, and backends have no run time limit. There's also a cronjob mechanism which can trigger requests at regular intervals.

You can fetch the data from external servers with the "URLFetch" service.

Upvotes: 5

Related Questions