CODEWITHSUNDEEP
elasticsearch
bigfoot
bigfoot

Reputation: 81

Elasticsearch is not working after full disk

I made space, restarted the service (no good) and then rebooted but I still get this in the elasticsearch.log:

[2015-02-16 13:35:19,625][WARN ][cluster.action.shard     ] [Server] [logstash-2015.02.16][1] sending failed shard for [logstash-2015.02.16][1], node[PFamB-ZJS7CwSdyyAcP_8A], [P], s[INITIALIZING], indexUUID [tZ3I9HZ6TDaZSicIuGWRWQ], 
    reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[logstash-2015.02.16][1] 
    failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [83]]; ]]
[2015-02-16 13:35:19,625][WARN ][cluster.action.shard     ] [Server] [logstash-2015.02.16][1] received shard failed for [logstash-2015.02.16][1], node[PFamB-ZJS7CwSdyyAcP_8A], [P], s[INITIALIZING], indexUUID [tZ3I9HZ6TDaZSicIuGWRWQ], 
    reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[logstash-2015.02.16][1] 
    failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [83]]; ]]
[2015-02-16 13:35:43,570][DEBUG][action.index             ] [Server] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-16 13:36:10,757][DEBUG][action.index             ] [Server] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]

What should I do?

Upvotes: 8

Views: 7673

Answers (2)

aakashrs
aakashrs

Reputation: 61

Bigfoot's solution is the only one that seems to work.

While the stack trace observed seems to be similar to: https://github.com/elastic/elasticsearch/issues/12055

This pull request is supposed to fix the issue: https://github.com/elastic/elasticsearch/pull/9797

But upgrading to v1.5.0 also does not do the trick.

Thus the only thing that works:

find  /var/lib/elasticsearch/elasticsearch/nodes/ -name "*.recovering"

And delete all recovering files. Of course this has side-effects.

Upvotes: 6

Brimstedt
Brimstedt

Reputation: 3140

I got the same problem too, with "No version type match" error in logs.

(Also, after disk got full)

Tried bigfoot's solution to manually delete recovery files and it worked.

Probably this can have some negative side effects though.

Upvotes: 0

Related Questions