CODEWITHSUNDEEP
elasticsearchlogstash
Abdelwahab Chaachoui
Abdelwahab Chaachoui

Reputation: 39

How to upload data to an existing index with Logstash?

I am trying to insert data from a .csv-File to an already existing index (that already has data) using Logstash.

Anyway this is my logstash_true.config File:

input {
    file {
        path => "pathToFile"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter {
    csv {
        separator => ","
        columns => ["title", "text", "subject", "date"]
    }

}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => "127.0.0.1:9200"
        index => "news"
        document_type => "true_news"
        document_id => "%{id}"
    }
}

When uploading the data, I can see in the command line that there is nothing wrong with the file or the Data and the document_type true_news actually exist.

But when trying the get the data:

{
  "count" : 0,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

The data wasn't loaded.

UPDATE

when enabling debugging i get the following error:

Could not index event to Elasticsearch. {:status=>400, :action=>
["index", {:_id=>"%{id}", :_index=>"news", :routing=>nil, 
:_type=>"true_news"}, #<LogStash::Event:0x7e10d60f>], :response=>
{"index"=>{"_index"=>"news", "_type"=>"true_news", "_id"=>"%{id}", 
"status"=>400, "error"=>{"type"=>"illegal_argument_exception", 
"reason"=>"Rejecting mapping update to [news] as the final mapping 
would have more than 1 type: [fake_news, true_news]"}}}}

Upvotes: 1

Views: 833

Answers (1)

leandrojmp
leandrojmp

Reputation: 7463

Since Elasticsearch version 6.0 you can't have multiple types in your index.

It seems that your index news already have documents or mapping with the type fake_news and you are trying to insert documents with the type true_news, this is not possible, that's why you are getting this error:

"type"=>"illegal_argument_exception", 
"reason"=>"Rejecting mapping update to [news] as the final mapping 
would have more than 1 type: [fake_news, true_news]"

Since you can have only 1 type and you want to be able to distinguish between true_news and fake_news, it is better to recreate your index to use the default type, doc, for every document, and add a tag with true_news or fake_news to your documents using the config add_tag => ["tag"] in your inpus.

Upvotes: 1

Related Questions