CODEWITHSUNDEEP
elasticsearchlogstash
Ramandeep Kaur
Ramandeep Kaur

Reputation: 21

how to fetch nested object matching the criteria from elastic using logstash

I am trying to retrieve nested object using nested datatype from elastic search using logstash with following config file named "export-nested.conf"

input {
 elasticsearch {
    hosts => "localhost:9200"
    index => "test"
    query => '
  {"query": {
      "nested": {
      "path": "comments",
      "query": {
        "match": {"comments.active": true}
      },
      "inner_hits": {
         "highlight": {
          "fields": {
            "comments.active": {}
          }
        }
      }
    }
}}'
  }
}
output {
   csv {
    fields => ["comments.author","comments.number"]
    path => "output.csv"
  }
}

To reproduce the issue: Step1:- I created below index with following mapping

PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "comments": {
          "type": "nested"
        }
      }
    }
  }
}

step2:- Entered data in the index that i created:

PUT test/_doc/1?refresh
{
  "title": "Test1",
  "comments": [
    {
      "author": "elis",
      "number": 1,
      "active": true
    },
    {
      "author": "zara",
      "number": 2,
      "active": false
    }
  ]
}

PUT test/_doc/2?refresh
{
  "title": "Test2",
  "comments": [
    {
      "author": "john",
      "number": 3,
      "active": false
    },
    {
      "author": "rob",
      "number": 4,
      "active": true
    }
  ]
}

Step3:- Used following command to run logstash

bin/logstash -f export-nested.conf

Output: I am getting blank data in the output file.

,
,

Expected output:

elis,1
rob,4

Upvotes: 0

Views: 573

Answers (1)

Ramandeep Kaur
Ramandeep Kaur

Reputation: 21

I finally got the resolution for the above query after reading the tutorials and spending a lot of time. I changed my logstash configuration file to resolve this. I have tested this and it is giving me the desired output.

input {
 elasticsearch {
    hosts => "localhost:9200"
    index => "objectindex"
    query => '
      {"query": {
        "match": {"comments.active": true}
      }}'
  }
}
filter {
   split {
     field => "comments"
   }
}
output {
  if [comments][active] {  
    stdout { codec => rubydebug }
    csv {
      fields => ["[comments][author]","[comments][number]"]
      path => "output.csv"
    }
  }
}

Output:-

elis,1
rob,4

Here, I have splitted the comments array using filter and then exporting the data of only those objects whose comments.active is true.

This configuration can be used with nested object "comments" of default datatype "object" and in the output plugin, I am printing it to the console as well as csv file. So, you can choose both or modify it as per your requirement.

-Thanks

Upvotes: 2

Related Questions