CODEWITHSUNDEEP
elasticsearch
user98651
user98651

Reputation: 324

Elasticsearch document assosiations

Is there a way in Elasticsearch to associate two documents to basically say: "These two belong together".

For example, I want to be able to associate TCP packets that are part of the same stream, so if I get a SYN and a SYN-ACK, I want to say that these two packets are related to each other.

Finally, I wish to be able to say: "Give me all the SYN packets which dont have a corresponding SYN-ACK".

Any help is appreciated

EDIT: Perhaps feeding the packets through a logstash aggregator would achieve this? Then I could group the packets into a "Stream index"??

Upvotes: 0

Views: 28

Answers (1)

Val
Val

Reputation: 217254

If you're loading your TCP events through Logstash, you can leverage the aggregate filter.

For instance, the filter configuration below will create a synthetic event on every SYN and "release" it on every SYN-ACK. You just need to identify a field some_unique_id that will bind both packets together as being related. Also make sure to see a proper timeout (here 120 seconds)

 filter {    
   if [type] == "SYN" {
     aggregate {
       task_id => "%{some_unique_id}"
       map_action => "create"
     }
   }

   if [type] == "SYN-ACK" {
     aggregate {
       task_id => "%{some_unique_id}"
       map_action => "update"
       end_of_task => true
       timeout => 120
     }
   }
 }

Upvotes: 1

Related Questions