CODEWITHSUNDEEP
elasticsearchkibanamonitoring
Korjavin Ivan
Korjavin Ivan

Reputation: 449

Got elasticsearch monitoring broken in kibana

One day my Kibana started to show that monitoring is switched off and it can't switch it on again.

enter image description here

In kibana's logs there is such messages:

{"type":"log","@timestamp":"2019-04-07T06:35:58Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":1,"message":"Uploading bulk stats payload to the local cluster"}
Unhandled rejection [export_exception] Exception when closing export bulk :: {"path":"/_xpack/monitoring/_bulk","query":{"system_id":"kibana","system_api_version":"6","interval":"10000ms"},"body":"{\"index\":{\"_type\":\"kibana_stats\"}}\n{\"kibana\":{\"uuid\":\"c1f81ebc-9b7f-47c0-90ce-47b44725ad25\",\"name\":\"kibana\",\"index\":\".kibana\",\"host\":\"0\",\"transport_address\":\"0:5601\",\"version\":\"6.6.1\",\"snapshot\":false,\"status\":\"green\"},\"usage\":{\"kql\":{\"optInCount\":1,\"optOutCount\":0,\"defaultQueryLanguage\":\"default-lucene\"},\"index\":\".kibana\",\"dashboard\":{\"total\":2},\"visualization\":{\"total\":3},\"search\":{\"total\":5},\"index_pattern\":{\"total\":7},\"graph_workspace\":{\"total\":0},\"timelion_sheet\":{\"total\":0},\"spaces\":{\"available\":true,\"enabled\":true,\"count\":3},\"apm\":{\"has_any_services\":false,\"services_per_agent\":{}},\"infraops\":{\"last_24_hours\":{\"hits\":{\"infraops_hosts\":0,\"infraops_docker\":0,\"infraops_kubernetes\":0,\"logs\":0}}},\"rollups\":{\"index_patterns\":{\"total\":0},\"saved_searches\":{\"total\":0},\"visualizations\":{\"total\":0,\"saved_searches\":{\"total\":0}}},\"xpack\":{\"reporting\":{\"available\":true,\"enabled\":true,\"browser_type\":\"chromium\",\"_all\":0,\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0},\"printable_pdf\":{\"available\":false,\"total\":0},\"status\":{},\"lastDay\":{\"_all\":0,\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0},\"printable_pdf\":{\"available\":false,\"total\":0},\"status\":{}},\"last7Days\":{\"_all\":0,\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0},\"printable_pdf\":{\"available\":false,\"total\":0},\"status\":{}}}}},\"cloud\":{\"name\":\"gcp\",\"id\":\"7164592281880898911\",\"vm_type\":\"custom-6-25600\",\"region\":\"us-central1\",\"zone\":\"us-central1-c\"},\"concurrent_connections\":0,\"os\":{\"load\":{\"1m\":0.61181640625,\"5m\":0.49462890625,\"15m\":0.46142578125},\"memory\":{\"total_in_bytes\":26338160640,\"free_in_bytes\":1914593280,\"used_in_bytes\":24423567360},\"uptime_in_millis\":1684000,\"platform\":\"linux\",\"platformRelease\":\"linux-4.14.91+\",\"distro\":\"Centos\",\"distroRelease\":\"Centos-7.6.1810\"},\"process\":{\"event_loop_delay\":1.137755999341607,\"memory\":{\"heap\":{\"total_in_bytes\":200716288,\"used_in_bytes\":160904432,\"size_limit\":1526909922},\"resident_set_size_in_bytes\":290590720},\"uptime_in_millis\":1209623},\"requests\":{\"disconnects\":0,\"total\":6},\"response_times\":{\"average\":131,\"max\":131},\"timestamp\":\"2019-04-07T06:35:57.046Z\"}\n","statusCode":500,"response":"{\"took\":10,\"ignored\":false,\"errors\":true,\"error\":{\"type\":\"export_exception\",\"reason\":\"Exception when closing export bulk\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"failed to flush export bulks\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"bulk [default_local] reports failures when exporting documents\",\"exceptions\":[{\"type\":\"export_exception\",\"reason\":\"RemoteTransportException[[elasticsearch-master-1][10.8.2.14:9300][indices:admin/create]]; nested: IllegalArgumentException[Rejecting mapping update to [.monitoring-kibana-6-2019.04.07] as the final mapping would have more than 1 type: [item, doc]];\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Rejecting mapping update to [.monitoring-kibana-6-2019.04.07] as the final mapping would have more than 1 type: [item, doc]\"}}]}}}}"}

I think that the most important part is "Rejecting mapping update to [.monitoring-kibana-6-2019.04.07] as the final mapping would have more than 1 type: [item, doc]"

Supposedly it happened after we had a performance issue and elasticsearch-client node restarted.

How could I fix this?

UPD: https://gist.github.com/korjavin/2c7031b83b777b27a7a4b4a63590b556

templates for kibana and es

Upvotes: 0

Views: 3916

Answers (1)

Korjavin Ivan
Korjavin Ivan

Reputation: 449

Finally we found the problem. One developer created a template with index match like index_patterns: ['*'] And it broke kibana monitoring.

Upvotes: 1

Related Questions