CODEWITHSUNDEEP
elasticsearch
Marcel Wolf
Marcel Wolf

Reputation: 326

Elasticsearch running in docker on VM - Index Crashes every hours

My setup:

elasticsearch logstash and metricbeat running as containers in docker-swarm mode. Metricbeat sents its logs to logstash and is written into the elasticsearch database.

One of each parts.

The Database persists on the host with the

--mount

parameter.

After some time (about 1h or earlyer). The index from metricbeat turns red.

and i got the log on elasticsearch:

[email protected] | [2017-09-01T08:26:08,287][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[metricbeat-2017.09.01][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[metricbeat-2017.09.01][0]] containing [125] requests]"})

[email protected] | [2017-09-01T08:26:08,287][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125}

I adjusted with template that there should only be one shard without replicas, but it did not help.

I tried a configuration with three different nodes, but still the same. i even tried different combinations of shard / replica count but not effect there.

_cat/shards shows:

metricbeat-2017.09.01 0 p STARTED 2998 1.6mb 10.143.14.2 tPPhaKt

Upvotes: 1

Views: 523

Answers (1)

Marcel Wolf
Marcel Wolf

Reputation: 326

I tried Not to Expose the Data Directory to the host and it works Fine. I think this was some I/O Problem with the Storage behind.

Upvotes: 0

Related Questions