CODEWITHSUNDEEP
hadoopmapreducecloudera
matt
matt

Reputation: 73

Hadoop streaming api - limit number of mappers on a per job basis

I have a job running on a small hadoop cluster that I want to limit the number of mappers it spawns per datanode. When I use the -Dmapred.map.tasks=12, it still spawns 17 mappers for some reason. I've figured out a way to limit it globally, but I want to do it on a per job basis.

Upvotes: 2

Views: 499

Answers (1)

Sourav Gulati
Sourav Gulati

Reputation: 1359

In Map Reduce , the total number of mappers will be spawned depends upon the input splits that are being created from your data .

There will be one mapper task spawned per input split. SO , you cannot decrease the count of mapper in Map Reduce.

Upvotes: 4

Related Questions