CODEWITHSUNDEEP
hadoopjarheap-memory
Ammar Sahib
Ammar Sahib

Reputation: 59

Increasing Java heap size for a class inside a jar

I need to execute java programs that are contained in a JAR file but I get an error that requires increasing the java heap size. I need to increase the heap size using command line. I tried java -Xmx6144M -d64 but it did not work. On the other hand, java -Xmx6144M ClassName requires ClassName. How can specify the class name that is inside a jar? is there some command to increase the java heap for all the classes?

Here is the error:

ava.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.regex.Pattern.compile(Pattern.java:1047)
    at java.lang.String.replace(String.java:2180)
    at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
    at org.apache.hadoop.fs.Path.initialize(Path.java:137)
    at org.apache.hadoop.fs.Path.<init>(Path.java:126)
    at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.<init>(RawLocalFileSystem.java:375)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:359)
    at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:290)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:721)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:746)
    at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:465)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:721)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:746)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:212)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:120)

Upvotes: 1

Views: 772

Answers (1)

Magham Ravi
Magham Ravi

Reputation: 603

I believe you need to increase the memory allocation for the launched map tasks. The default is 200 MB which seems to be insufficient in your case. If the driver class is a subclass of Tool, you can pass the following arguments -Dmapred.child.java.opts=-Xmx500m . Here , you are bumping up the memory from 200 to 500.

You can also do profiling of your map reduce job to see which objects are taking up memory.

Upvotes: 2

Related Questions