CODEWITHSUNDEEP
javahadoopmapreducebigdataheap-memory
pythonic
pythonic

Reputation: 21635

How to increase the heap size when using hadoop jar?

I am running a program with the hadoop jar command. However, to make that program run faster, I need to increase Hadoop's heap size. I tried the following, but it didn't have any effect (I have hadoop version 2.5), even on the same terminal.

export HADOOP_HEAPSIZE=16384
export HADOOP_OPTS="-Xmx16g"

Another way I reckon is to add the following to the mapred-site.xml file, but I am unfortunately not the admin, so can't do it.

<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx16384m</value>
</property>

Is there any other method as well to do this?

Upvotes: 0

Views: 2677

Answers (2)

PALEN
PALEN

Reputation: 2876

A way to increase the heap size when using Hadoop jar that works for me:

HADOOP_CLIENT_OPTS="-XX:-UseGCOverheadLimit -Xmx16384m" hadoop jar parquetExample.jar

as you can see you set the environment variable's value (works for other environment variables too, in this case I am setting the max size to 16384m) in the same line and before the hadoop jar command.

Upvotes: 1

pythonic
pythonic

Reputation: 21635

I solved this problem by modifying the HADOOP_CLIENT_OPTS environment variable, as shown below.

export HADOOP_CLIENT_OPTS="-Xmx16g $HADOOP_CLIENT_OPTS"

Note that the program I am using runs only on the Master node. In other words, its not a distributed application.

Upvotes: 2

Related Questions