CODEWITHSUNDEEP
elasticsearchlucenejvmsearch-engine
Utkarsh Mishra
Utkarsh Mishra

Reputation: 490

Elasticsearch 1.5.2 High JVM Heap in 2 nodes even without bulk indexing

We have been facing multiple downtimes recently, especially after few hours of Bulk Indexing. To avoid further downtimes, we disabled bulk indexing temporarily and added another node. Now downtimes have stopped but two out 6 nodes permanently remain at JVM Heap > 80%

We have a 6 node cluster currently (previously 5), each being EC2 c3.2xlarge with 16 GB ram, 8 GB of JVM heap, all master+data. We're using Elasticsearch 1.5.2 which has known issues like [OOM Thrown on merge thread](https://issues.apache.org/jira/browse/LUCENE-6670 OOM Thrown on merge thread), and we faced the same regularly.

There are two major indices used frequently for Search and autosuggest having doc count/size as follows:

health status index pri rep docs.count docs.deleted store.size
green open aggregations 5 0 16507117 3653185 46.2gb
green open index_v10 5 0 3445495 693572 44.8gb

Ideally, we keep at least one replica for each index, but our last attempt to add replica with 5 nodes resulted in OOM errors and full heap, so we turned it back to 0. We also had two bulk update jobs running between 12-6 AM, each updating about 3 million docs with 2-3 fields on a daily basis. They were scheduled at 1.30 AM and 4.30 AM, each sending bulk feed with 100 docs (about 12 KB in size) to BULK api using a bash script having a sleep time of .25s between each to avoid too many parallel requests. When we started the bulk update, we had max 2 million docs to update daily, but the doc count almost doubled in a short span (to 3.8 million) and we started seeing Search response time spikes mostly between 4-6 AM and sometimes even later. Our average Search response time also increased from 60-70 ms to 150+ ms. A week ago, master left due to ping timeout, and soon after that we received shard failed error for one index. On investigating further, we found that this specific shard's data was inaccessible. To save unavailability of data, we restarted the node and reindexed the data.

However, the node downtime happened many more times, and each time Shards went into UNASSIGNED or INITIALIZING state. We finally deleted the index and started fresh. But heavy indexing again brought OutOfMemory Errors and node downtime, with same shard issue and data loss. To avoid further downtimes, we stopped all bulk jobs and reindexed data at a very slow rate.

We also added one more node to distribute load. Yet, currently we have 3 nodes with JVM constantly above 75+%, 2 being 80+ always. We have noticed that number of segments and their size is relatively high on these nodes (about 5 GB), but using optimize index on these would risk increasing heap again, with a probability of downtime.

Another important point to note is that our tomcat apps hit only 3 of all nodes (for normal search and indexing), and mostly one of the other two node was used for bulk indexing. Thus, out of three query+indexing receiving node, one node, and the left node for bulk indexing has relatively high heap.

There are following known issues with our configuration and indexing approach, which are planning to fix:

  1. Bulk indexing hits only one node, thus increasing its heap, and causes slightly high GC pauses.
  2. mlockall is set to false
  3. Snapshot is needed to revert index in such cases, we're under planning phase when this incident happened.
  4. We can merge 2 bulk jobs into one, to avoid too indexing request under queue at the same time.
  5. We can use optimize API at regular interval in the bulk indexing script to avoid existence of too many segments.

Elasticsearch yml: (only relevant and enabled settings mentioned)

master: true index.number_of_shards: 5 index.number_of_replicas: 2

path.conf: /etc/elasticsearch path.data: /data

transport.tcp.port: 9300 transport.tcp.compress: false

http.port: 9200 http.enabled: true

gateway.type: local gateway.recover_after_nodes: 2 gateway.recover_after_time: 5m gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 4 # Now that we have 6 nodes discovery.zen.ping.timeout: 3s

discovery.zen.ping.multicast.enabled: false

Node stats: Pastebin link Hot threads: Pastebin link

Upvotes: 3

Views: 176

Answers (1)

silvestrelosada
silvestrelosada

Reputation: 55

If I understand well you have 6 servers, each of one of them is running elasticsearch node.

What I would to is to run more than one node on each server separating the roles, node that act as client, node that acts as data node, and node that act as master. I think that you can have two nodes on each server.

3 servers: data + client

3 servers: data + master

The client nodes and the master nodes will need less amount of RAM. The configuration files will be more complex but it will work better.

Upvotes: 0

Related Questions