Apache Solr intermittently shuts down

Question

Occasionally my Apache Solr will just go down and this is spammed in the log files:

Sep 24, 2013 1:00:21 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

The dedicated server my Solr install is on has plenty of free RAM (16gb), what can be causing this? I've tried some Google Fu but with no solid answer as to why, or how to fix beyond using -Xmx512mb to raise the allowed heap size which I don't think is the most ideal method? I could be wrong.

I'm using Jetty with my Solr install, is there a way to tell Jetty to use more memory? or to tell it how to resolve this error should it happen again.

Extra Info: The thing is if I set any large Xms or Xmx values, even if it's within my free memory left on the server I get There is insufficient memory for the Java Runtime Environment to continue.

#  Out of Memory Error (allocation.inline.hpp:58), pid=10851, tid=47145744939328
#
# JRE version: 7.0_11-b21
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

VM Arguments:
jvm_args: -Dsolr.solr.home=/opt/solr/solr -Xmx64m -Djetty.logs=/var/log/solr -Djetty.home=/root/wmv_solr -Djava.io.tmpdir=/tmp -Xmx256m 
java_command: /root/wmv_solr/start.jar /root/wmv_solr/etc/jetty-logging.xml /root/wmv_solr/etc/jetty.xml
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/java/jdk1.7.0_11
CLASSPATH=.:/usr/java/jdk1.7.0_11/lib/classes.zip
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/java/jdk1.7.0_11/bin:/opt/ant/bin:/usr/local/bin:/usr/X11R6/bin:/root/bin
SHELL=/bin/bash

/proc/meminfo:
MemTotal:      4033216 kB
MemFree:        755528 kB
Buffers:        274004 kB
Cached:        1939244 kB
SwapCached:     168388 kB
Active:        1923800 kB
Inactive:      1052876 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      4033216 kB
LowFree:        755528 kB
SwapTotal:     4980024 kB
SwapFree:      4784124 kB
Dirty:            2092 kB
Writeback:           0 kB
AnonPages:      762616 kB
Mapped:          39044 kB
Slab:           256200 kB
PageTables:      17496 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   6996632 kB
Committed_AS:  1964148 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    265936 kB
VmallocChunk: 34359471775 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

This is still an ongoing issue and we've been seeing a new kind of error in logs:

Oct 02, 2013 7:14:10 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded

We're looking at upgrading our servers RAM in hopes that this will resolve our issue, but it doesn't explain why SOLR isn't using the otherwise free RAM.

Answer 1

From the documentation link

Memory usage in solr:

• All of the document's fields must simultaneously fit into memory.
• Each individual ... tag in the input XML must fit into memory

Solving out of memory exception

• The easiest way to fight this error, assuming the Java virtual machine isn't already using all your physical memory, is to increase the amount of memory allocated to the Java virtual machine running Solr. To do this for the example/ in the Solr distribution, if you're running the standard Sun virtual machine, you can use the -Xms and -Xmx command-line parameters:

  java -Xms512M -Xmx1024M -jar start.jar
  

Play with the values depending on your input