appengine instance hours calculation

Question

I develop an appengine application right now in python and I am surprised by the instance hours quotas I get, while I try to optimize my app for costs and performance.

I am testing right now one specific task_queue. (nothing else is running during that - before I start no instance is up)

the queue is configured with a rate/s of 100 with 100 buckets. no configured limit for max_concurrent_requests

900 tasks will get pushed in this queue. 10-11 instances pop-up in this moment to deal with it.

everything takes far less than 30 seconds and every task is executed.

I check my instance hours quotas before and after that and I consume about 0.25 - 0.40 instance hours.

why is that?

shouldn't it be much less? is there an inital cost or a minimum amount which will be charged if one instance opens?

Answer 1

I am pretty sure that the Scheduler will increase instance count when there is a backlog of tasks on a high-rate queue. 100/100 is a very high rate. You are telling the Scheduler to do these very quickly which means it fires up instances to do so.

Unless you need to process these tasks very quickly, you should use a much lower rate. This will result in fewer instances, and a longer queue of tasks. Depending on your processing requirements, you might be able to use a pull queue. Doing so allows you to lease and process hundreds of tasks at a time and take advantage of batch put()s etc. Really depends on what you are doing.

Answer 2

When an instance is opened it will cost you at least 15 minutes. Your 10-11 instances should cost you a total of around 2.5 hours.
If you don't need such a fast processing you should limit the amount of parallel processing of the queue using max_concurrent_requests.