CODEWITHSUNDEEP
opensearch
user2173353
user2173353

Reputation: 4660

OpenSearch not starting - [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];

After indexing some documents, my OpenSearch is not starting. In the opensearch.log file, I get this error:

org.opensearch.cluster.block.ClusterBlockException: index [.opensearch-observability] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
    at org.opensearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:243) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:221) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:95) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:56) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:237) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onNewClusterState(TransportClusterManagerNodeAction.java:336) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:230) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.9.0.jar:2.9.0]
    at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.9.0.jar:2.9.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]

I have emptied my disk a bit (Win11 system) and I now have 64GB free, but the error won't go away.

I've found some related ElasticSearch questions, but I don't think I can use any of the solutions, given that I can't even start OpenSearch: Elasticsearch read_only_allow_delete auto setting

BTW, my free disk storage space is at 16%. For ElasticSearch they say that 15% is required to be free, in my provided link.

Any ideas on what can be done?

Update :
I have freed up more space and my disk has now 20% free space. Nothing changes with OpenSearch...

Update #2 :
I have freed up more space and my disk has now 25% free space. Nothing changes with OpenSearch...

Update #3 :
I have moved the OpenSearch folder to an external disk with 75% free space. Nothing changes with OpenSearch... This is insane! What is wrong with it?

Update #4 :
The full logs: https://mega.nz/file/tMsDRDjQ#AxDAfuv_0A5wj1qdUmLhbPyzQksqLCdnfJNusxba3EQ

Upvotes: 0

Views: 444

Answers (1)

Silky Sandpaper
Silky Sandpaper

Reputation: 11

Its a hard reset - but it worked.

Make sure opensearch is stopped. My instance was dead in the water at this stage, even after I increased disk partitions. (See growfs, df -h, lsblk -f, diskpart)

systemctl status opensearch

If not - stop it

systemctl stop opensearch

Stretch/grow/increase HD partitions if you can to get more space, or all of this would be futile.

Move data around

cd /var/lib/opensearch
mv nodes/ nodes_old/
systemctl start opensearch

Wait for Opensearch to start up (tail -f /var/log/opensearch/*.log), and then stop it again to clear any blocks. At this stage we are duping Opensearch into a clean slate instance.

systemctl stop opensearch

Move the data back

mv nodes_old/ nodes/

Start Opensearch again

systemctl start opensearch

Monitor Opensearch (tail -f /var/log/opensearch/*.log)

Upvotes: 1

Related Questions