CODEWITHSUNDEEP
elasticsearchlogstash
KaranGulati
KaranGulati

Reputation: 1

Aggregation in Logstash-ElasticSearch

I am using logstash with input-elasticsearch and output-elasticsearch.Both Elastic Search have a different instance. Before the data goes to the output block,I want to aggregate some documents,create a hash of the new document and insert the nested document in the elastic search. So basically I want to do some processing before nested document is inserted in the elasticsearch.Is this possible?

input{
    # something here to get a value of variable stored in a different file
    elasticsearch{
    hosts=>"abc.de.fg.hi:jklm"
    query=>'{--some query---}'
    }
    }
    output{

    elasticsearch{
    hosts=>"xxx.xx.xx.xx:yyyy"

    }

Upvotes: 0

Views: 1747

Answers (1)

Jony Feldman
Jony Feldman

Reputation: 96

I'm using the "aggregate" plug in.

In my case the input is From UDP and i filter it with "grok" but i believe you can achieve what you want to do by tweaking the code a bit.

Without a sample of you are trying to achieve exactly, the best this i can do is show you a sample of my code:

aggregate {
    task_id => “%{action}_%{progress}”
    code =>
    “
    map[‘avg’] || = 0;
    map[‘avg’] += event.get(‘elapsed’);
    map[‘my_count’] || = 0;
    map[‘my_count’] += 1;

    if (map[‘my_count’] == ${LogstashAggregationCount})#Environment variable
        event.set(‘elapsedAvg’, (map[‘avg’] / map[‘my_count’]))
        event.set(‘Aggregetion’, true)
        map[‘avg’] = 0
        map[‘my_count’] = 0
    end
    “
}    
if (![Aggregetion]) {
    drop {}
}

Of curse you need to adapt it to your specific case. For more in depth explanation of my code read here: How to Use Logstash Aggregations

Upvotes: 2

Related Questions