CODEWITHSUNDEEP
javaamazon-web-servicesmemorymapreduceemr
intl
intl

Reputation: 2773

AWS EMR - There is insufficient memory for the Java Runtime

I am running a MapReduce job on AWS EMR. The map job completes except for one file that is very large. I get the following error:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000611280000, 1521483776, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1521483776 bytes for committing reserved memory.

It seems to be a memory issue. I've modified my configuration json file to have added (a lot more than required) memory:

[
  {
    "Classification": "hadoop-env",
    "Properties": {},
    "Configurations": [
      {
        "Classification": "export",
        "Properties": {
           "HADOOP_DATANODE_HEAPSIZE": "10240",
           "HADOOP_NAMENODE_OPTS": "-XX:GCTimeRatio=19",
           "HADOOP_HEAPSIZE": "11264",
           "HADOOP_CLIENT_OPTS": "-Xmx10240M"
       }
      }
    ]
  },
  {
    "Classification": "mapred-site",
    "Properties": {
      "mapreduce.map.memory.mb": "24576",
      "mapreduce.map.java.opts": "-Xmx19200M",
      "mapred.child.java.opts": "-Xmx4096M",
      "mapreduce.reduce.memory.mb": "15360",
      "mapreduce.reduce.java.opts": "-Xmx10240M",
      "mapreduce.job.jvm.numtasks": "1",
      "mapreduce.job.reuse.jvm.num.tasks": "1"
    }
  },
  {
    "Classification": "yarn-site",
    "Properties": {
      "yarn.scheduler.maximum-allocation-mb": "25600",
      "yarn.nodemanager.resource.memory-mb": "25600"
    }
  },
  {
    "Classification": "hive-env",
    "Properties": {}
  },
  {
    "Classification": "hive-site",
    "Properties": {}
  }
]

However, I keep on getting the issue. As you can see, I have added mapred.child.java.opts as many suggest online, but I've had no luck. What else can I try?

Much appreciated.

Upvotes: 1

Views: 6355

Answers (1)

ChristopherB
ChristopherB

Reputation: 2068

It appears your configuration is exceeding the physical memory bounds of the server. The m3.xl only physically has 15G and by default the safe amount of memory to allocate to a container is 11.5G (http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-hadoop-task-config.html).

So for the m3.xl the largest you can set mapreduce.map.java.opts is -Xmx9216 with mapreduce.map.memory.mb at 11520 (the opts should always be less than total map memory, usually around 80%). These are the properties that impact the map task memory size. If the map task needs more memory in order to process the larger files, then a larger instance type will need to be used.

I recommend not making other memory property changes unless those processes are needing such tuning specifically.

Upvotes: 3

Related Questions