CODEWITHSUNDEEP
elasticsearch
Sudhir Maurya
Sudhir Maurya

Reputation: 51

One of elasticsearch node keep going to error state

I am trying to run an elastic search cluster of 2 nodes, but one of the nodes keeps going to Error state while other node working perfectly fine. Below are the logs

Kubernetes version 1.16.1
elasticsearch version elasticsearch-oss:6.4.3

es-cluster-0                                   1/1     Running   0          66s

es-cluster-1                                   0/1     Error     2          66s

[ERROR][o.e.g.GatewayMetaState   ] [es-cluster-1] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:3, file:/usr/share/elasticsearch/data/nodes/0/indices/00hAD3unTn-m80CsNnk5YA/_state/state-3.st]
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:200) ~[elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:64) ~[elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:86) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.node.Node.<init>(Node.java:447) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.node.Node.<init>(Node.java:256) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.3.jar:6.4.3]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.3.jar:6.4.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.3.jar:6.4.3]
Caused by: java.io.IOException: failed to read [id:3, file:/usr/share/elasticsearch/data/nodes/0/indices/00hAD3unTn-m80CsNnk5YA/_state/state-3.st]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.4.3.jar:6.4.3]
        ... 14 more

Caused by: java.lang.IllegalArgumentException: Unexpected field [mapping_version]
        at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1324) ~[elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1372) ~[elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1361) ~[elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.3.jar:6.4.3]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.3.jar:6.4.3]
        ... 14 more

Upvotes: 0

Views: 1273

Answers (1)

Dave Turner
Dave Turner

Reputation: 1896

This index was created or modified by Elasticsearch version 6.5.0 or later, and cannot be read in this version any more. This is usually because you upgraded Elasticsearch and then tried to downgrade it again. Downgrades are not possible.

The simplest solution is to continue with the upgrade. If that's not possible, you will need to wipe this node and start again, restoring any lost data from a recent snapshot.

Upvotes: 1

Related Questions