CODEWITHSUNDEEP
hadoopmapreduce
HHH
HHH

Reputation: 6475

How to change the heap size of a Hadoop map task within the java code?

As the following link says, the default value for a map task's heap size is 200 MB, however I need to increase it because I need to increase the mapreduce.task.io.sort.mb to at least 300 or 400MB. I don't have access to Hadoop config files in order to make such a change, so I has to do it in my code. How can I do that?

http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

Upvotes: 2

Views: 7935

Answers (2)

Amar
Amar

Reputation: 12020

In order to set/chnage hadoop configurations programmatically, you can have your driver class as follows:

    public class MyDriver extends Configured implements Tool {


        @Override
        public int run(String[] args) throws Exception {

                Configuration conf = getConf();

                conf.set("mapred.child.java.opts", "-Xmx1024m -Xss600m");
                conf.set("mapreduce.task.io.sort.mb", "400m");
...

    }
    }

Upvotes: -1

user1261215
user1261215

Reputation:

We can set the parameters through Configuration API.

Configuration conf = new Configuration();
conf.set("mapred.child.java.opts", "heap size here");
Job job = new Job(conf);

Also, mapred.child.ulimit should be 2–3x higher than the heap size specified in mapred.child.java.opts

Upvotes: 5

Related Questions