CODEWITHSUNDEEP
elasticsearchlogstash
Anthony Richir
Anthony Richir

Reputation: 649

Create nested object if not exists in a Logstash pipeline

In a Logstash pipeline, I'm trying to verify if a nested object exists. If it doesn't, I want to create it and initialized its fields.

Here is my pipeline output:

output {
    elasticsearch {
        hosts => ["${ELASTICSEARCH_HOST:localhost:9200}"]
        user => "${ELASTICSEARCH_USERNAME:elastic}"
        password => "${ELASTICSEARCH_PASSWORD:password}"
        index => "shape_idx"
        action => "update"
        document_id => "%{shapeId}"
        script_type => "inline"
        scripted_upsert => true
        document_type => "shape"
        script => '    
            if (ctx._source["realViews"]) {
                ctx._source.realViews.all=0;
                ctx._source.realViews.desktop=0;
                ctx._source.realViews.phone=0;
                ctx._source.realViews.tablet=0;
            }
            ctx._source.realViews.all = (ctx._source.realViews.all ?:0) + params.event.deviceCounter.all;
            ctx._source.realViews.desktop = (ctx._source.realViews.desktop ?:0) + params.event.deviceCounter.desktop;
            ctx._source.realViews.phone = (ctx._source.realViews.phone ?:0) + params.event.deviceCounter.phone;
            ctx._source.realViews.tablet = (ctx._source.realViews.tablet ?:0) + params.event.deviceCounter.tablet;
        '
    }
}

Unfortunately, my pipeline fails silently, any idea ? For the record, I'm using Logstashg 7.1.1 and Elasticsearch 5.6.16.

Upvotes: 0

Views: 232

Answers (1)

YouryDW
YouryDW

Reputation: 393

Mutations should happen in a mutate filter, just add a check there and add the field when needed within your filter.

Try to remind yourself that you have input / filter / output and make each part do what it is supposed to do.

Let the input just input data in a pipeline, maybe add a field if you really want to (useful if you use multiple inputs in the same pipeline running through the same filter).

Let the filter do whatever manipulation you want to do, this includes checking if fields exist and then acting upon it.

Let the output just output (if statements to have conditional outputs can be part of it, but no further processing).

Upvotes: 1

Related Questions