CODEWITHSUNDEEP
elasticsearch
V-K
V-K

Reputation: 1347

ElasticSearch fails with OutOfMemoryError

I have green ES cluster with 3 nodes. All worked well, but recent times there are some fails.

[2019-04-22T11:05:37,099][WARN ][o.e.t.OutboundHandler    ] [node_1] send message failed [channel: Netty4TcpChannel{localAddress=/172.0.0.1:9300, remoteAddress=/172.0.0.2:41674}]
java.nio.channels.ClosedChannelException: null
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2019-04-22T11:05:37,096][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node_1] fatal error in thread [elasticsearch[node_1][search][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:302) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:311) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.automaton.Automaton$Builder.addTransition(Automaton.java:715) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.automaton.UTF32ToUTF8.all(UTF32ToUTF8.java:247) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.automaton.UTF32ToUTF8.end(UTF32ToUTF8.java:231) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.automaton.UTF32ToUTF8.build(UTF32ToUTF8.java:194) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.automaton.UTF32ToUTF8.convertOneEdge(UTF32ToUTF8.java:137) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:307) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:230) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:104) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:81) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.search.WildcardQuery.<init>(WildcardQuery.java:67) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
        at org.apache.lucene.queryparser.classic.QueryParserBase.newWildcardQuery(QueryParserBase.java:644) ~[lucene-queryparser-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:17:41]
        at org.apache.lucene.queryparser.classic.QueryParserBase.getWildcardQuery(QueryParserBase.java:703) ~[lucene-queryparser-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:17:41]
        at org.elasticsearch.index.search.QueryStringQueryParser.getWildcardQuerySingle(QueryStringQueryParser.java:682) ~[elasticsearch-6.7.1.jar:6.7.1]

and there are some questions:

  1. As you can see in logs wildcard query is last(2 times), does it means ES fails during the wildcard or it can be just coincidence?
  2. If yes, what can be a reason that fails? Can the wrong search query cause such error?

Upvotes: 0

Views: 865

Answers (1)

Gabriel
Gabriel

Reputation: 302

As lendrojmp said, wildcard can use lot of memory, especially if it's start with *.

According to documentation:

In order to prevent extremely slow wildcard queries, a wildcard term should not start with one of the wildcards * or ?. The wildcard query maps to Lucene WildcardQuery.

https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-wildcard-query.html

You can also check the gc.log.xx file in your log directory (default /var/log/elasticsearch). You may have more insight, check also the slow queries logs.

Check also about this post: https://www.elastic.co/blog/found-crash-elasticsearch

You may run out of resources for another reason and the wildcard search take the left memory and crash your server.

Upvotes: 1

Related Questions