CODEWITHSUNDEEP
javaelasticsearchdebianout-of-memory
Ruydo
Ruydo

Reputation: 305

Query Failed [Failed to execute main query] OutOfMemoryError: Java heap space

Hi i have data in elastic 80 M of data have 52 Go and i search in text with this query

$json_doc = '{
                 "query": {
                    "query_string": {
                        "query": "*' . $sujet . '*",
                        "lenient": true
                    }
                    },
     "size": 10000000,
     "from": 0,
     "fields": [
     "association_key","text"
                    ]
                }';

when I search a word that is in the base is looking less 20K I have my data but as his exceeds 20 K I have this error

[2015-11-14 20:03:04,663][DEBUG][action.search.type       ] [Pixx] [myindex3][2], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][2]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:56)
    at org.apache.lucene.search.HitQueue.<init>(HitQueue.java:64)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:275)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:37)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:135)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:133)
    at org.apache.lucene.search.TopScoreDocCollector.create(TopScoreDocCollector.java:262)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:490)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more
[2015-11-14 20:03:11,764][DEBUG][action.search.type       ] [Pixx] [myindex3][0], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][0]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:187)
    at org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:81)
    at org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
    at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
    at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:144)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more

how i can resolve this problem i have dedicate server contains just a Intel Xeon W3520 c 4/8 t + 2.66 GHz RAM 32 GB ECC under Debian

Upvotes: 2

Views: 1675

Answers (2)

Alain O&#39;Dea
Alain O&#39;Dea

Reputation: 21686

UPDATE: ES_HEAP_SIZE is no longer a documented option

Elastic Search won't use all of that memory without being configured.

You need to configure and tune Elastic Search and Debian's memory settings.

You'll need to configure one of these:

  • /etc/elasticsearch/jvm.options: a file containing JVM options (file is at config/jvm.options (in a TAR or ZIP distro rather than a Debian package)
  • ES_JAVA_OPTS: an environment variable with additional JVM settings including memory and GC tuning if needed (pretty much what's in jvm.options but in a single line separated by spaces)

Here's the relevant excerpt from Elastic's latest documentation as of February 2020:

Here are examples of how to set the heap size via the jvm.options file:

-Xms2g ①
-Xmx2g ②
  • ① Set the minimum heap size to 2g.
  • ② Set the maximum heap size to 2g.

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS:

ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch ①
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch ②
  • ① Set the minimum and maximum heap size to 2 GB.
  • ② Set the minimum and maximum heap size to 4000 MB.

See more here:

Upvotes: 2

su-ex
su-ex

Reputation: 106

Seems like you don't have enough space for Java's heap. Since it seems like you'd have more hardware wise you may try to increase it: Increase heap size in Java

Upvotes: 1

Related Questions