CODEWITHSUNDEEP
google-app-engine
Andrew W
Andrew W

Reputation: 85

How to renew leases with Pull Queue tasks?

I have some long-lived tasks that may take more than a week of work to complete (parts of the pipeline can only work during certain hours of the day).

Renewing leases seems like the right approach, since the current max lease length is 1wk. Renewing leases is mentioned briefly in one overview doc, but I don't find any other documentation for how to use this.

There are some other oblique references to it - the taskqueue module docs have an exception that can occur when you try and fail to renew a lease...but no method for renewing a lease. There is a doc on migrating to Cloud Tasks that mentions a task.renewLease method, but says it doesn't apply to folks using the built-in GAE TaskQueue support.

Upvotes: 1

Views: 114

Answers (1)

Alex
Alex

Reputation: 5276

I've only ever used push queues, but modify_task_lease() may do what you need: https://cloud.google.com/appengine/docs/standard/python/refdocs/google.appengine.api.taskqueue#google.appengine.api.taskqueue.Queue.modify_task_lease

It was talked about here: https://cloud.google.com/appengine/docs/standard/python/taskqueue/pull/leasing-pull-tasks

Alternatively, given that you said this:

(parts of the pipeline can only work during certain hours of the day).

You may want to consider braking up these tasks into bite-size stages that are safe to repeat, possibly using the datastore to save state.

I discovered a memory-leak in app engine's python ssl library one time, which caused my whole instance to crash every couple minutes and it really messed up my long running tasks, as they would all just restart.

Upvotes: 4

Related Questions