CODEWITHSUNDEEP
pythongoogle-app-enginetask-queue
Ragnar
Ragnar

Reputation: 543

AppEngine Task Queue API Calls increase on TaskAlreadyExistsError

I'm using deferred to put tasks in the default queue in a AppEngine app similar to this approach.

Im naming the task with a timestamp that changes every 5 second. During that time a lot of calls are made to the queue with the same name resulting in a TaskAlreadyExistsError which is fine. The problem is when I check the quotas the "Task Queue API Calls" are increasing for every call made, not only those who actually are put in the queue.

I mean if you look at the quota: Task Queue API Calls: 34,017 of 100,000 and compare to the actual queue calls: /_ah/queue/deferred - 2.49K

Here is the code that handles the queue:

try:
  deferred.defer(function_call, params, _name=task_name, _countdown=int(interval/2))
except (taskqueue.TaskAlreadyExistsError, taskqueue.TombstonedTaskError):
  pass

I suppose that is the way it works. Is there a good way to solve the problem with the quota? Can I use memcache to store the task_name and check if the task has been added besides the above try/catch? Or is there a way to check if the task already exists without using Task Queue Api Calls?

Upvotes: 2

Views: 816

Answers (2)

Eren Güven
Eren Güven

Reputation: 2374

The "good way" to solve the quota problem is eliminating destined-to-fail calls to Task Queue API.

_name you are using changes every 5 seconds which might not be a bottleneck if you increase the execution rate of your Task Queue. But you also add Tasks using _countdown.

Upvotes: 0

Steven Kampen
Steven Kampen

Reputation: 631

Thanks for pointing out that this is the case, because I didn't realise, but the same problem must be affecting me.

As far as I can see yeah, throwing something into memcache comprised of the taskname should work fine, and then if you want to reduce those hits on memcache you can store the flag locally also within the instance.

Upvotes: 2

Related Questions