CODEWITHSUNDEEP
pythongoogle-app-enginequeuetask
Will Curran
Will Curran

Reputation: 7110

How do Google App Engine Task Queues work?

I'm confused about Task execution using queues. I've read the documentation and I thought I understood bucket_size and rate, but when I send 20 Tasks to a queue set to 5/h, size 5, all 20 Tasks execute one after the other as quickly as possible, finishing in less than 1 minute.

deferred.defer(spam.cookEggs, 
               egg_keys, 
               _queue="tortoise")  

- name: tortoise
  rate: 5/h  
  bucket_size: 5

What I want is whether I create 10 or 100 Tasks, I only want 5 of them to run per hour. So it would take 20 Tasks approximately 4 hours to complete. I want their execution spread out.

UPDATE

The problem was I assumed that when running locally, that Task execution rate rules were followed, but that is not the case. You cannot test execution rates locally. When I deployed to production, the rate and bucket size I had set executed as I expected.

Upvotes: 26

Views: 1617

Answers (2)

Charles Duffy
Charles Duffy

Reputation: 295403

Execution rates are not honored by the app_devserver. This issue should not occur in production.

[Answer discovered by Nick Johnson and/or question author; posting here as community wiki so we have something that can get marked accepted]

Upvotes: 7

rmmh
rmmh

Reputation: 7095

You want to set bucket_size to 1, or else you'll have "bursts" of queued activity like you saw there.

From the documentation:

bucket_size

Limits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution rate. For example, consider a queue with a rate of 5/s and a bucket size of 10. If that queue has been inactive for some time (allowing its "token bucket" to fill up), and 20 tasks are suddenly enqueued, it will be allowed to execute 10 tasks immediately. But in the following second, only 5 more tasks will be able to be executed because the token bucket has been depleted and is refilling at the specified rate of 5/s.

Upvotes: -1

Related Questions