CODEWITHSUNDEEP
elasticsearchlogstashlogstash-configuration
Navarro
Navarro

Reputation: 1384

Logstash and Elastic upgrade

I had a functional Logstash and Elasticsearch on version 5.1.

I deleted all indices, then upgraded to 6.1.

Now, when Logstash receives some event from Filebeat (Which stills version 5.1), it throws this error:

[2017-12-27T17:29:16,463][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch.
{
  :status => 400,
  :action => ["index", {:_id=>nil, :_index=>"logstash-2017.12.27", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x34de85bd>],
  :response => {
    "index" => {
      "_index" => "logstash-2017.12.27",
      "_type" => "doc",
      "_id" => nil,
      "status" => 400,
      "error" => {
        "type" => "mapper_parsing_exception",
        "reason" => "Failed to parse mapping [_default_]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.",
        "caused_by" => {
          "type" => "mapper_parsing_exception",
          "reason" => "[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."
        }
      }
    }
  }
}

I have even tried using an extremely simplistic pipeline, as you can see here:

input {
  beats {
    port => 5044
  }
}   

filter {
    json {
      source => "message"
    }   
}   

output {
  elasticsearch { hosts => ["localhost:9200"] }
}

Yet it throws this error over and over.

Any idea what can be wrong here?

Upvotes: 2

Views: 1384

Answers (2)

Wjdavis5
Wjdavis5

Reputation: 4151

This answer is to just expand on what @alexanderlz said. From the DevTools page in kibana I ran this:

GET /_template/

That lists all templates

here is the template we need to delete / modify (in part):

"logstash": {
    "order": 0,
    "version": 60001,
    "index_patterns": [
      "logstash-*"
    ],

So then run

DELETE /_template/logstash

once that is done restart logstash and it will reinstall a new, correct, template.

Upvotes: 2

alexanderlz
alexanderlz

Reputation: 589

take a look at changes in mapping, introduced in elasticsearch 6.0

you need to remove the include_in_all mapping parameter from your index template.

can you paste here your template/mapping?

Upvotes: 1

Related Questions