Kibana / Elasticsearch: Connecting from kibana to es fails with no shards available exception

Question

I am trying to connect via kibana to an es cluster (1 master - 1 data).

Kibana front end gives 504 error.

In my kibana logs there are no errors.

However in es:

[2019-02-22T11:39:33,764][WARN ][r.suppressed             ] path: /.kibana/doc/config%3A6.4.2/_update, params: {refresh=wait_for, index=.kibana, id=config:6.4.2, type=doc}
org.elasticsearch.action.UnavailableShardsException: [.kibana][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update
    at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.retry(TransportInstanceSingleOperationAction.java:211) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.doStart(TransportInstanceSingleOperationAction.java:166) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$2.onTimeout(TransportInstanceSingleOperationAction.java:232) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:573) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.2.jar:6.4.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2019-02-22T11:39:33,768][WARN ][r.suppressed             ] path: /.kibana/doc/config%3A6.4.2/_update, params: {refresh=wait_for, index=.kibana, id=config:6.4.2, type=doc}
org.elasticsearch.action.UnavailableShardsException: [.kibana][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update
    at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.retry(TransportInstanceSingleOperationAction.java:211) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.doStart(TransportInstanceSingleOperationAction.java:166) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$2.onTimeout(TransportInstanceSingleOperationAction.java:232) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:573) [elasticsearch-6.4.2.jar:6.4.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.2.jar:6.4.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

I have tried to delete the .kibana index and restart all services with no luck.

curl -XGET http://master01-elastic:9200
{
  "name" : "master01",
  "cluster_name" : "local-stg-cluster",
  "cluster_uuid" : "K3zb-E6xRle7MWjYrag4nA",
  "version" : {
    "number" : "6.4.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "04711c2",
    "build_date" : "2018-09-26T13:34:09.098244Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}


curl -XGET http://master01-elastic/_cat/allocation?v
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
     2                                                                   UNASSIGNED

 curl -XGET http://master01-elastic.dev.encode.local:9200/_cat/indices?v
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana K3NoYmDaRnGk9vem8oUFlQ   1   1   


curl -XGET http://master01-elastic:9200/_cat/shards?v
index   shard prirep state      docs store ip node
.kibana 0     p      UNASSIGNED               
.kibana 0     r      UNASSIGNED  


curl -XGET 'http://master01-elastic.dev.encode.local:9200/_recovery?human&detailed=true&active_only=true'
{}

$ curl -XGET 'http://master01-elastic.dev.encode.local:9200/_cluster/allocation/explain'
{"index":".kibana","shard":0,"primary":true,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2019-02-22T16:36:40.852Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes"}

Answer 1

When troubleshooting this kind of problems (i.e. UNASSIGNED shards), it first helps to know whether the allocation failed for some reason during the node recovery by running this:

curl -XGET 'localhost:9200/_recovery?human&detailed=true&active_only=true'

In your case, the response was empty which means that it wasn't a recovery problem.

Sometimes, if a shard failed to be allocated too many times, it will stay unassigned until you run this command:

curl -XPOST http://master01-elastic/_cluster/reroute?retry_failed=true

If that doesn't help, the next step consists of checking the allocation decisions to see if anything stands out, by running this:

curl -XGET http://master01-elastic/_cluster/allocation/explain

In your case, that yields this:

{
  "index": ".kibana",
  "shard": 0,
  "primary": true,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "INDEX_CREATED",
    "at": "2019-02-22T16:36:40.852Z",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes"
}

This can be the case if your data node is down or if you have some cluster-level or index-level shard allocation filtering rules (like preventing the shards of a given index to be allocated to a given node). You can see if that's the case by checking the settings of your cluster, respectively your index.

curl -XGET http://master01-elastic/.kibana/_settings
curl -XGET http://master01-elastic/_cluster/settings

Check if you have something in the index.routing.allocation.* section (for index-level rules)...

"settings": {
  "index": {
    "routing": {
      "allocation": {
        "include": {
          "_name": "NODE1,NODE2"
        },
        "exclude": {                        <--- this might be the issue
          "_name": "NODE3,NODE4"
        }
      }
    },

...or in the cluster.routing.allocation.* section (for cluster-level rules)

"cluster": {
  "routing": {
    "allocation": {
      "enable": "none"                      <--- this might be the issue
    }
  }

If that's the case, then you might have to adapt your rules.