CODEWITHSUNDEEP
elasticsearch
curious1
curious1

Reputation: 14727

low disk watermark [??%] exceeded on

I use Elasticsearch 1.4.4 in my development machine (a single notebook). Everything is set as default because I never changed any settings.

When I start it, I usually get the following message: 

[2015-10-27 09:38:31,588][INFO ][node                     ] [Milan] version[1.4.4], pid[33932], build[c88f77f/2015-02-19T13:05:36Z]
[2015-10-27 09:38:31,588][INFO ][node                     ] [Milan] initializing ...
[2015-10-27 09:38:31,592][INFO ][plugins                  ] [Milan] loaded [], sites []
[2015-10-27 09:38:34,665][INFO ][node                     ] [Milan] initialized
[2015-10-27 09:38:34,665][INFO ][node                     ] [Milan] starting ...
[2015-10-27 09:38:34,849][INFO ][transport                ] [Milan] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.81.1.108:9300]}
[2015-10-27 09:38:35,022][INFO ][discovery                ] [Milan] elasticsearch/DZqnmWIZRpapZY_TPkkMBw
[2015-10-27 09:38:38,787][INFO ][cluster.service          ] [Milan] new_master [Milan][DZqnmWIZRpapZY_TPkkMBw][THINKANDACT1301][inet[/10.81.1.108:9300]], reason: zen-disco-join (elected_as_master)
[2015-10-27 09:38:38,908][INFO ][http                     ] [Milan] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.81.1.108:9200]}
[2015-10-27 09:38:38,908][INFO ][node                     ] [Milan] started
[2015-10-27 09:38:39,220][INFO ][gateway                  ] [Milan] recovered [4] indices into cluster_state
[2015-10-27 09:39:08,801][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node
[2015-10-27 09:39:38,798][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node
[2015-10-27 09:40:08,801][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node
....

I see a lot of these "low disk watermark ... exceeded on..." messages. What went wrong in my case? How to fix it? Thanks!

UPDATE

Before this post, I searched SO for related posts. I found one related to "high watermark..." and in that case, the disk space is low. In my case, I checked and there is still 56GB left on my disk.

UPDATE

According to the input from Andrei Stefan, I need to change settings. Should I do it the following way:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'

Or is there any settings file I can edit to set it?

Upvotes: 81

Views: 148065

Answers (6)

Sergey Shcherbina
Sergey Shcherbina

Reputation: 1

on Mac OS config path - /opt/homebrew/etc/elasticsearch/elasticsearch.yaml

just add

  cluster.routing.allocation.disk.threshold_enabled: true 
  cluster.routing.allocation.disk.watermark.flood_stage: 200mb
  cluster.routing.allocation.disk.watermark.low: 500mb 
  cluster.routing.allocation.disk.watermark.high: 300mb

Upvotes: 0

sastorsl
sastorsl

Reputation: 2135

If you like me have a lot of disk you can tune the watermark setting and use byte values instead of percentages:

NB! Use either percentage values or byte values.

You cannot mix the usage of percentage/ratio values and byte values within the watermark settings. Either all values are set to percentage/ratio values, or all are set to byte values.

Setting: cluster.routing.allocation.disk.watermark.low

Controls the low watermark for disk usage. It defaults to 85%, meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can alternatively be set to a ratio value, e.g., 0.85. It can also be set to an absolute byte value (like 500mb) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices but will prevent their replicas from being allocated.

Setting: cluster.routing.allocation.disk.watermark.high

Controls the high watermark. It defaults to 90%, meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can alternatively be set to a ratio value, e.g., 0.9. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not.

Setting:: cluster.routing.allocation.disk.watermark.flood_stage

Controls the flood stage watermark, which defaults to 95%. Elasticsearch enforces a read-only index block (index.blocks.read_only_allow_delete) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark. Similarly to the low and high watermark values, it can alternatively be set to a ratio value, e.g., 0.95, or an absolute byte value.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation

Please note:

Percentage values refer to used disk space, while byte values refer to free disk space. This can be confusing, since it flips the meaning of high and low. For example, it makes sense to set the low watermark to 10gb and the high watermark to 5gb, but not the other way around.

On my 5TB disk I've set:

# /etc/elasticsearch/elasticsearch.yml
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 30gb
cluster.routing.allocation.disk.watermark.high: 20gb

Upvotes: 105

Ivan Yurchenko
Ivan Yurchenko

Reputation: 3871

I added the following lines to my elasticsearch.yaml file (elastic_search_folder\config):

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.disk.watermark.high: 95%

For some reason, specifying watermark in gb like in the other answer didn't work for me. Also, make sure your watermark.high is less or equal to the flood watermark (it's usually set to 95%).

This worked for me in version 6.1.1.

Upvotes: 14

Amit
Amit

Reputation: 32386

I want to add some background on why Elasticsearch has these various thresholds and what can happen if these threshold crossed.

Background

Elasticsearch considers the available disk space before deciding whether to allocate new shards, relocate shards away or put all indices on read mode based on a different threshold of this error. The reason is Elasticsearch indices consists of different shards which are persisted on data nodes.

Solution

My detailed blog on relevant settings and temporary and permanent fix can be found here.

In short, permanent fixes are:

  1. Delete unused indices.
  2. Merge segments using the answers from Optimize API for reducing the segments and eliminating ES deleted docs not working.
  3. Attach external disk or increase the disk used by the problematic data node.

Upvotes: 0

iyanglow
iyanglow

Reputation: 399

In my case - I had just to turn off threshold:

run ElasticSearch:

elasticsearch

On other tab run:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'


curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

macOS Catalina, ElasticSearch installed via Brew.

Upvotes: 29

Milad Hamid Elhaei Sahar
Milad Hamid Elhaei Sahar

Reputation: 583

I know it is old post, but my comment can make someone happy. In order to specify watermark in bytes values (gb or mb) you have to add cluster.routing.allocation.disk.watermark.flood_stage to your elasticsearch settings file - elasticsearch.yml. Complete example: 

  cluster.routing.allocation.disk.threshold_enabled: true 
  cluster.routing.allocation.disk.watermark.flood_stage: 200mb
  cluster.routing.allocation.disk.watermark.low: 500mb 
  cluster.routing.allocation.disk.watermark.high: 300mb

Note: without specifying cluster.routing.allocation.disk.watermark.flood_stage it will not work with bytes value (gb or mb)

Upvotes: 48

Related Questions