CODEWITHSUNDEEP
elasticsearchlogstashcouchdb
Shekhar
Shekhar

Reputation: 387

Logstash to Elasticsearch adding new data in the fields instead of overwriting the existing data?

My pipeline is like this: CouchDB -> Logstash -> ElasticSearch. Every time I update a field value in CouchDB, the data in Elasticsearch is overwritten. My requirement is that, when data in a field is updated in CouchDB, I want to create a new data in Elasticsearch instead of overwriting the existing one.

My current logstash.conf is like this:

input {
    couchdb_changes {
        host => "<ip>"
        port => <port>
        db => "test_database"
        keep_id => false
        keep_revision => true
        initial_sequence => 0
        always_reconnect => true
        #sequence_path => "/usr/share/logstash/config/seqfile"
    }
}

output {
    if([doc][doc_type] == "HR") {
        elasticsearch {
            hosts => ["http://elasticsearch:9200"]
            index => "hrindex_new_1"
            document_id => "%{[doc][_id]}"
            user => elastic
            password => changeme
        }
    }
    if([doc][doc_type] == "SoftwareEngg") {
        elasticsearch {
            hosts => ["http://elasticsearch:9200"]
            index => "softwareenggindex_new"
            document_id => "%{[doc][_id]}"
            user => elastic
            password => changeme
        }
    }
}

How to do this?

Upvotes: 0

Views: 790

Answers (1)

leandrojmp
leandrojmp

Reputation: 7463

You are using the document_id option in your elasticsearch output, what this option does is tell elasticsearch that it should index the document using this value as the document id, which will be a unique id.

document_id => "%{[doc][_id]}"

So, if in your source document the field [doc][_id] has for example the value of 1000, the _id field in the elasticsearch will also have the same value.

When you change something in the source document that has the [doc][_id] equals to 1000, it will replace the document with the _id equals to 1000 in elasticsearch because the _id is unique.

To achieve what you want, you will need to remove the option document_id from your outputs, this way elasticsearch will generate a unique value for the _id field of the document.

elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "softwareenggindex_new"
    user => elastic
    password => changeme
}

Upvotes: 3

Related Questions