CODEWITHSUNDEEP
elasticsearchkibanakibana-4elasticsearch-indices
Orz
Orz

Reputation: 615

ElasticSearch entered "read only" mode, node cannot be altered

Something happened during the night to my ES cluster (composed of 5 data nodes, 3 master nodes).

I have no idea what happened but all the indices and data were deleted and the cluster entered a "read only" mode, possibly hacked?

When trying to get Kibana running, I get the following: kibana

Tried restarting Kibana - it restarted, nothing changed. Tried restarting Elastic - it restarted (all nodes), nothing changed.

I then had a look at the cluster settings and this is what I got:

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      },
      "blocks": {
        "read_only": "true"
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      }
    }
  }
}

I tried undoing the read only as follows:

PUT _cluster/settings
{
  "persistent": {
    "blocks.read_only": false
  }
}

No luck as you can see:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status": 403
}

Any ideas?

UPDATE: Problem solved by Andrei Stefan, now for the more important part - why? What happened and why? I've lost all data and my cluster entered a read-only mode.

Upvotes: 12

Views: 16368

Answers (2)

dg6
dg6

Reputation: 480

It turns out ES has some thresholds for available disk space, and when the "flood" one is hit, it puts the indeces into read only mode.

In order to set it back (tested with ES6), you will need to do the following:

PUT /[index_name]/_settings
{
  "index.blocks.read_only_allow_delete": null
}

More information can be found on the following page of the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

Upvotes: 12

Andrei Stefan
Andrei Stefan

Reputation: 52368

The correct command is:

PUT /_cluster/settings
{
  "persistent" : {
    "cluster.blocks.read_only" : false
  }
}

Upvotes: 17

Related Questions