CODEWITHSUNDEEP
pythongoogle-app-enginemapreduce
maciekrb
maciekrb

Reputation: 227

How to run appengine mapreduce jobs in a module?

I've been reviewing the new "Modules API" in appengine, and trying to organize the app in a way that the most demanding tasks are routed through the dispatch.yaml to more powerful instances through module configurations.

However, when attempting to configure "/mapreduce/" uris in the dispatcher, they never seem to run in the correct module, they always run in the default module.

The dispatcher looks like this : 

application: simple-sample
dispatch:
  - url: "*/mapreduce/*"
    module: bigger-instance

  - url: "*/_ah/pipeline/*"
    module: bigger-instance

The modules are defined as the documentation states, default in app.yaml and bigger-instance in bigger-instance.yaml containing the mapreduce handler entries, but so far every mapreduce/* process executes in the default handler according to the logs, getting a 404, since the handlers are defined elsewhere.

Any ideas ?

Upvotes: 2

Views: 786

Answers (1)

theScarletManuka
theScarletManuka

Reputation: 96

The simplest option is to set the target property of your mapreduce pipeline.

pipeline = mapreduce.mapreduce_pipeline.MapreducePipeline( ... )
pipeline.with_params(target="version-dot-module")
pipeline.start()

This target is passed on to the task queues and overrides the task queue default routing which you were experiencing. If there is no matching version of the specified module then the default version of the module is used. You may specify just "version", in which case the default module applies. Specifying a non-existent module will target the default version of the default module, even if there is a matching version of the default module.

It is also possible to define a target for a selected task queue then route your pipeline to that queue. This approach has the limitation that the routing applies for all versions of your calling code, but perhaps there is virtue in keeping the routing details in the yaml files instead of the codebase.

Upvotes: 3

Related Questions