CODEWITHSUNDEEP
hadoop
rafique haque
rafique haque

Reputation: 1

Increasing io.sort.mb

It would be highly appreciated if someone could help me to find out what went wrong in my configuration.

I wanted to increase the value of io.sort.mb and thus I added the property below in core-site.xml. io.sort.mb 350m

The runtime information I am attaching below clearly shows that the value of io.sort.mb, did not change rather the default value io.sort.mb = 100 stayed. 

13/08/15 16:43:34 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1e5e96c1
13/08/15 16:43:34 INFO mapred.MapTask: numReduceTasks: 1
13/08/15 16:43:34 INFO mapred.MapTask: **io.sort.mb = 100**
13/08/15 16:43:34 INFO mapred.MapTask: data buffer = 79691776/99614720
13/08/15 16:43:34 INFO mapred.MapTask: record buffer = 262144/327680
13/08/15 16:43:34 INFO mapred.MapTask: Starting flush of map output
13/08/15 16:43:34 INFO mapred.MapTask: Finished spill 0
13/08/15 16:43:34 INFO mapred.Task: Task:attempt_local_0001_m_004609_0 is done. And is in the process of commiting

Since it was not working, I added the property in mapred-site.xml schema, however I got the same outcome as above.

Can anyone suggest me what should I do?

Thanking you in advance.

Haq

Upvotes: 0

Views: 7860

Answers (2)

linehrr
linehrr

Reputation: 1748

this conf should be in mapred-site.xml instead of core-site.xml

refer: http://hadoop.apache.org/docs/r1.0.4/mapred-default.html

Upvotes: 0

on_the_shores_of_linux_sea
on_the_shores_of_linux_sea

Reputation: 1022

according to the article here io.sort.mb should be 10 * io.sort.factor incase you have ram.
"core-site.xml"

<property>
<name>io.sort.factor</name>
<value>100</value>
<description>More streams merged at once while sorting files.</description>
</property>  

<property>
<name>io.sort.mb</name>
<value>200</value>
<description>Higher memory-limit while sorting data.</description>
</property>

trying changing sort factor also on all nodes.

Upvotes: 1

Related Questions