CODEWITHSUNDEEP
google-app-enginemapreduce
JasonC
JasonC

Reputation: 349

legacy mapreduce library only running one shard

We are using the "legacy" (non-PipelineAPI) version of the mapreduce library: http://code.google.com/p/appengine-mapreduce/

The issue is that we can only ever get one shard processing, even for kinds that have >150,000 entities. We have tried different shard_count configurations, e.g, 4, 16, 128, but always only one shard processing entire dataset, which is very slow.

I feel like I've missed a step (e.g., creating an index or something). We just this legacy library successfully on other applications and it runs with multiple shards as expected.

Crossing my fingers that someone knows an offhand answer.

Thanks, j

Upvotes: 1

Views: 101

Answers (1)

JasonC
JasonC

Reputation: 349

Found it!

This issue occurs in Python and only when you override the default kind name for your model.

I have put a patch for this on the mapreduce issue tracker: http://code.google.com/p/appengine-mapreduce/issues/detail?id=154

Upvotes: 2

Related Questions