Task queue filled

Question

I use the task queue to send users future reminders. Each task is very small, but somehow this situation has happened and my app is down:

Any ideas what might cause this?

Update

Just ~30 minutes later I'm getting a very different report.

"Tasks in queue" count is almost the same, but "task queue stored task count" is suddenly much less. :o

Just to be clear what I did:

• I purged the "deleter" task, which was using a lot of CPU but had only 16 small tasks in it, so it shouldn't have really affected the size of the queue much.

• Caught the exception that happens when you try to add a task to the full queue. This shouldn't affect amount of tasks added at all.

• Browsed Reddit for half an hour. That fixed it.

Answer 1

I had this problem but I was queueing up several thousand tasks in a short time. If you feel nothing is wrong in your code then just increase the quota for Task Queue stored task bytes. Include these lines in your queue.yaml to change the quota

total_storage_limit: 300M

Answer 2

Sounds like you've got a "fork bomb" in your task queue code.

If you add tasks within a task you can accidentally create a huge number of tasks if an exception is being raised after the task is added. The original task gets retried, the new task gets added again (and possibly does its own bomb), the exception gets raised again, the task gets retried, etc.

One way to prevent this is to give the new task a specific name so that it can't be added to the queue twice.

Brett Slatkin talks about this in one of his Google I/O lectures (here at 7:58).