CODEWITHSUNDEEP
javagoogle-app-enginemapreducetask-queue
Riley Lark
Riley Lark

Reputation: 20890

How do I throttle appengine-mapreduce on AppEngine?

http://code.google.com/p/appengine-mapreduce/ mentions that it can control the speed of execution, but I can't figure out how. It would make sense to create a task queue for a mapper job and control the speed from there, but I don't see how to specify which queue to use.

Upvotes: 2

Views: 503

Answers (3)

autumngard
autumngard

Reputation: 225

For users using python, you may want to search for this variable known as "_DEFAULT_PROCESSING_RATE_PER_SEC" in model.py to increase the default processing rate per second. That worked for me!

Upvotes: 1

Casey
Casey

Reputation: 398

(Thanks to Chris for the pointer, but it took a bit of digging after that to find the exact parameter.)

From http://code.google.com/p/appengine-mapreduce/wiki/UserGuideJava add this to your mapreduce.xml configuration element to limit the number of entities processed per second:

<property>
    <name>mapreduce.mapper.inputprocessingrate</name>
    <value>4</value>
</property>

Upvotes: 2

Chris Farmiloe
Chris Farmiloe

Reputation: 14175

I don't use the Java version myself, but the Python version has a processing_rate param that you pass to the mapper spec.

A quick search in the Java source reveals a MAPPER_INPUT_PROCESSING_RATE_KEY config key, which hopefully points you in the right direction.

Upvotes: 2

Related Questions