Reputation: 2773
I am running a MapReduce job on AWS EMR. The map job completes except for one file that is very large. I get the following error:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000611280000, 1521483776, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1521483776 bytes for committing reserved memory.
It seems to be a memory issue. I've modified my configuration json file to have added (a lot more than required) memory:
[
{
"Classification": "hadoop-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"HADOOP_DATANODE_HEAPSIZE": "10240",
"HADOOP_NAMENODE_OPTS": "-XX:GCTimeRatio=19",
"HADOOP_HEAPSIZE": "11264",
"HADOOP_CLIENT_OPTS": "-Xmx10240M"
}
}
]
},
{
"Classification": "mapred-site",
"Properties": {
"mapreduce.map.memory.mb": "24576",
"mapreduce.map.java.opts": "-Xmx19200M",
"mapred.child.java.opts": "-Xmx4096M",
"mapreduce.reduce.memory.mb": "15360",
"mapreduce.reduce.java.opts": "-Xmx10240M",
"mapreduce.job.jvm.numtasks": "1",
"mapreduce.job.reuse.jvm.num.tasks": "1"
}
},
{
"Classification": "yarn-site",
"Properties": {
"yarn.scheduler.maximum-allocation-mb": "25600",
"yarn.nodemanager.resource.memory-mb": "25600"
}
},
{
"Classification": "hive-env",
"Properties": {}
},
{
"Classification": "hive-site",
"Properties": {}
}
]
However, I keep on getting the issue. As you can see, I have added mapred.child.java.opts as many suggest online, but I've had no luck. What else can I try?
Much appreciated.
Upvotes: 1
Views: 6355
Reputation: 2068
It appears your configuration is exceeding the physical memory bounds of the server. The m3.xl only physically has 15G and by default the safe amount of memory to allocate to a container is 11.5G (http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-hadoop-task-config.html).
So for the m3.xl the largest you can set mapreduce.map.java.opts is -Xmx9216 with mapreduce.map.memory.mb at 11520 (the opts should always be less than total map memory, usually around 80%). These are the properties that impact the map task memory size. If the map task needs more memory in order to process the larger files, then a larger instance type will need to be used.
I recommend not making other memory property changes unless those processes are needing such tuning specifically.
Upvotes: 3