google app engine modules - long running tasks > 10 minutes

Question

I am trying to port to google app engine modules, a previously long-running job ( running in 'backends' ) .

A sample module1.yaml is present below.

application: myuniqueapp
module:  module1
version: 1
runtime: python27
api_version: 1
threadsafe: true
instance_class: F4_1G
automatic_scaling:
  max_idle_instances: 1

handlers:
- url: /data
  static_dir: data
  application_readable: true

- url: /.*
  script: app.application

The code to submit to this (from a front-end instance), via taskqueue is this:

taskqueue.add(url='/tasks/do_my_task',
            target='1.module1')

This submits the right task without an issue. The task gets executed by the module1 as well.

But it gets killed by the 10th minute, with the DeadlineExceededError. This is a long running task and runs for longer than 10 minutes ( like how it used to work for 'backends' ).

What configuration change needs to be done, for the task executing in a module to be > 10 minutes ?

Answer 1

Google appengine also provides backinstance classes where a task can run upto 24 hours.Re-route the tasks request to run on backinstance which will solve the 10 min issue.Hope this will solve your issue

Answer 2

You have a few options, but currently there is no way of getting around the 10 minute request deadline for App Engine auto-scaling modules. Simply put, you need to make your task run faster, or you need to run it on a service that has no deadline.

Faster:

• Chunk the task into smaller processes and queue multiple tasks.
• Perform optimizations on the Task and make it run under 10 minutes.

No Deadline:

• Create another auto-scaling/manual app engine service to send the task
• Change your module from auto-scaling to manual or basic scaling.
• Create another auto-scaling/manual app engine service to send the task to.

Answer 3

You simply need to choose manual or basic scaling for your module:

https://developers.google.com/appengine/docs/java/modules/#Java_Instance_scaling_and_class