CODEWITHSUNDEEP
pythongoogle-app-enginecronmapreduce
Pawel Markowski
Pawel Markowski

Reputation: 1196

How to start mapreduce job from cron on GAE Python

I have mapreduce job defined in mapreduce.yaml:

mapreduce:
- name: JobName 
  mapper:
    input_reader: google.appengine.ext.mapreduce.input_readers.DatastoreInputReader
    handler: handler_name
    params:
    - name: entity_kind
      default: KindName

How to start it from cron? Is there some url that can run it?

Upvotes: 5

Views: 1060

Answers (2)

Chris Farmiloe
Chris Farmiloe

Reputation: 14185

You can start a mapreduce task from any kind of AppEngine handler using control.py

from mapreduce import control

mapreduce_id = control.start_map(
    "My Mapper",
    "main.my_mapper",
    "mapreduce.input_readers.DatastoreInputReader",
    {"entity_kind": "models.MyEntity"},
    shard_count=10)

Upvotes: 10

Chris Bunch
Chris Bunch

Reputation: 89823

Yes, if you look at the Getting Started page, it shows that you set the URL in your app.yaml:

handlers:
- url: /mapreduce(/.*)?
  script: mapreduce/main.py
  login: admin

You then can just cron it in the usual App Engine fashion, which in this example would be writing a cron.yaml like this:

cron:
- description: daily summary job
  url: /mapreduce
  schedule: every 24 hours

Upvotes: -6

Related Questions