CODEWITHSUNDEEP
elasticsearch
Eric G
Eric G

Reputation: 935

ElasticSearch Unassigned shards with two nodes( different machines), 1 master both new instances

I started up two clean elasticsearch instances (nodes), no data, two different machines ( one windows, one osx). They successfully discover each other. One is node.master: false. Both are node.data: true. I started Kibana ( creates te .kibana index ) and I created a test index (test), number_of_replicas=1 and the status of each index and the cluster is yellow which I believe is because of unassigned shards. I am at a loss how to get the unassigned shards to resolve.

In trying to force the replication of the shard I get the following error:

shard cannot be allocated on same node [tNUHIE6cTHO6h37P_s3m7w] it already exists on

Some details:

_cat/nodes?v:

 host         ip       heap.percent ram.percent  load node.role master name              
192.168.1.99 192.168.1.99     2          81  1.95 d         *      node1     
192.168.1.2  192.168.1.2     13          46 -1.00 d         -      node2

Node 1: _cluster/health

{
 "cluster_name": "elasticsearch",
 "status": "yellow",
 "timed_out": false,
 "number_of_nodes": 2,
 "number_of_data_nodes": 2,
 "active_primary_shards": 6,
 "active_shards": 9,
 "relocating_shards": 0,
 "initializing_shards": 0,
 "unassigned_shards": 3,
 "delayed_unassigned_shards": 0,
 "number_of_pending_tasks": 0,
 "number_of_in_flight_fetch": 0,
 "task_max_waiting_in_queue_millis": 0,
 "active_shards_percent_as_number": 75

}

There are no errors in the logs but if I run:

_cluster/reroute?pretty

{ "commands" : [ { "allocate" : { "index" : "test", "shard" : 1, "node" : "node2" } } ] 
}

I get the following response:

{ "error": {
  "root_cause": [
     {
        "type": "illegal_argument_exception",
        "reason": "[allocate] allocation of [test][1] on node { node2}{tNUHIE6cTHO6h37P_s3m7w}{192.168.1.2}{192.168.1.2:9300}{master=false} is not allowed, 
         reason: [YES(target node version [2.1.1] is same or newer than source node version [2.1.1])] 
         [YES(enough disk for shard on node, free: [111.6gb])]
         [YES(shard not primary or relocation disabled)]
         [YES(primary is already active)][YES(node passes include/exclude/require filters)]
         [YES(allocation disabling is ignored)]
         [NO(shard cannot be allocated on same node [tNUHIE6cTHO6h37P_s3m7w] it already exists on)]
         [YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(below shard recovery limit of [2])][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)]"
     }
     ],
  ...
  "status": 400
}

_cat/shards?v

  index   shard prirep state      docs store ip           node          
 test      3     p      STARTED       0  130b 192.168.1.2 node2 
 test      3     r      UNASSIGNED                                          
 test      4     r      STARTED       0  130b 192.168.1.2  node2
 test      4     p      STARTED       0  130b 192.168.1.99 node1   
 test      1     p      STARTED       0  130b 192.168.1.2 node2
 test      1     r      UNASSIGNED                                       
 test      2     r      STARTED       0  130b 192.168.1.2  node2
 test      2     p      STARTED       0  130b 192.168.1.99 node1
 test      0     r      STARTED       0  130b 192.168.1.2 node2
 test      0     p      STARTED       0  130b 192.168.1.99 node1     
 .kibana 0    p      STARTED       1 3.1kb 192.168.1.2  node2 
 .kibana 0    r      UNASSIGNED

Any help for a newbie would be apprecitated in resolving this.

Upvotes: 2

Views: 2819

Answers (1)

bittusarkar
bittusarkar

Reputation: 6357

You can only reroute replica shards safely. GET _cat/shards?v clearly shows that shard (primary) of id 1 of test index is already allocated on node2. You cannot allocate a shard on the same node it is already allocated on. That is exactly what the output of _cluster/reroute command is telling you. Instead of allocating on node2, allocate it on node1. Try the command below:

POST _cluster/reroute?explain
{
   "commands": [
      {
         "allocate": {
            "index": "test",
            "shard": 1,
            "node": "node1"
         }
      },
      {
         "allocate": {
            "index": "test",
            "shard": 2,
            "node": "node1"
         }
      }
   ]
}

This will try to allocate both the unassigned replica shards. Also note the explain option. The response of the command will give a verbose explanation as to why the commands succeeded or failed and that comes in very handy while debugging if the commands do fail.

Upvotes: 1

Related Questions