CODEWITHSUNDEEP
elasticsearchelasticsearch-5
Tamizharasan
Tamizharasan

Reputation: 293

Cluster Red - unallocated shards in a index

My cluster suddenly went to red. Because of an index shard allocation fail. when i run

GET /_cluster/allocation/explain
{
  "index": "my_index",
  "shard": 0,
  "primary": true
}

output: 

{
  "shard": {
    "index": "twitter_tracker",
    "index_uuid": "mfXc8oplQpq2lWGjC1TxbA",
    "id": 0,
    "primary": true
  },
  "assigned": false,
  "shard_state_fetch_pending": false,
  "unassigned_info": {
    "reason": "ALLOCATION_FAILED",
    "at": "2018-01-02T08:13:44.513Z",
    "failed_attempts": 1,
    "delayed": false,
    "details": "failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: NotSerializableExceptionWrapper[shard_lock_obtain_failed_exception: [twitter_tracker][0]: obtaining shard lock timed out after 5000ms]; ",
    "allocation_status": "no_valid_shard_copy"
  },
  "allocation_delay_in_millis": 60000,
  "remaining_delay_in_millis": 0,
  "nodes": {
    "n91cV7ocTh-Zp58dFr5rug": {
      "node_name": "elasticsearch-24-384-node-1",
      "node_attributes": {},
      "store": {
        "shard_copy": "AVAILABLE"
      },
      "final_decision": "YES",
      "final_explanation": "the shard can be assigned and the node contains a valid copy of the shard data",
      "weight": 0.45,
      "decisions": []
    },
    "_b-wXdjGRdGLEtvY76PDSA": {
      "node_name": "elasticsearch-24-384-node-2",
      "node_attributes": {},
      "store": {
        "shard_copy": "NONE"
      },
      "final_decision": "NO",
      "final_explanation": "there is no copy of the shard available",
      "weight": 0,
      "decisions": []
    }
  }
}

What will be the solution? This is happened in my production node. My elasticsearch version 5.0. and i have two nodes

Upvotes: 3

Views: 10608

Answers (2)

Aidar Gatin
Aidar Gatin

Reputation: 843

It is an issue that every Elastic Cluster developer will bump to anyway :)

Safeway to reroute your red index.

curl -XPOST 'localhost:9200/_cluster/reroute?retry_failed

This command will take some time, but you won't get allocation error while data transferring.

Here is issue explained wider.

Upvotes: 3

Tamizharasan
Tamizharasan

Reputation: 293

I solved my issue with the following command.

curl -XPOST 'localhost:9200/_cluster/reroute?pretty' -d '{
    "commands" : [ {
        "allocate_stale_primary" :
            {
              "index" : "da-prod8-other", "shard" : 3,
              "node" : "node-2-data-pod",
              "accept_data_loss" : true
            }
        }
    ]
}'

So here u might lose the data. For me it works well. It was thrilling while running the command. luckily it worked well. For more details enter link description here Check this thread.

Upvotes: 1

Related Questions