CODEWITHSUNDEEP
pythonelasticsearchhttp-postelasticsearch-plugin
Anurag Sharma
Anurag Sharma

Reputation: 5049

filter query on specific field in elastic search

I have a document like this 

  {
    _index: "logstash-2015.11.30",
    _type: "hadoopgeneric",
    _id: "AVFVsF6ypMu_z_qvIUgL",
    _score: null,
    _source: {
             @timestamp: "2015-11-30T00:00:00.017Z",
             message: "selector : 48 - Element found for using multiple selectors using query .js-product-brand.product-brand",
             @version: "1",
             host: "ip-x-x-x-x",
             path: "/logs/stats/container/application_1448508514184_0178/container_e06_1448508514184_0178_01_003568/stderr",
             type: "hadoopgeneric",
             thread_id: "15119",
             thread_name: "MainThread",
             component_name: "Page",
             severity: "DEBUG",
             env: "STG",
             role: "spider",
             ip: "x.x.x.x",
             tags: [
                 "processed"
             ]
            },
   }

I have to filter those documents that have path /logs/stats/container/application_1448508514184_0178/container_e06_1448508514184_0178_01_003568/stderr in it (in path field particular)

I tried this general search query http://localhost:9200/logstash-*/_search?pretty=true&q="/logs/stats/container/application_1448508514184_0178/container_e06_1448508514184_0178_01_003568/stderr"&sort=@timestamp&size=100000

Its giving me the result but now I am thinking of trying to search only in path field by firing this query like this (I am getting no results in this query) -- http://localhost:9200/logstash-*/_search?pretty=true&q="path: /logs/stats/container/application_1448508514184_0178/container_e06_1448508514184_0178_01_003568/stderr"&sort=@timestamp&size=100000

I was going through this document Term Query on elastic search. But I am not sure how to pass such queries as post parameter in elastic search. I am using python library to make a post request to elastic search

Below is what I have tried so far

esurl = http://localhost:9200/logstash-*/_search
r = requests.post(esurl,data={"term":{'path':'/logs/stats/container/application_1448508514184_0178/container_e06_1448508514184_0178_01_003568/stderr'}})
r.text

{"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[5D_RNDQPRf6xyLO1suIoCA][logstash-2015.11.30][0]: RemoteTransportException[[ip-x-x-x-x-elkstorage][inet[/x.x.x.x:9300]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[logstash-2015.11.30][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [_na_]]]; nested: ElasticsearchParseException[Failed to derive xcontent]; }{[o8jLb8P5SWOfsCo78eUlHg][logstash-2015.12.01][0]: RemoteTransportException[[ip-x-x-x-x-elkstorage][inet[/x.x.x.x:9300]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[logstash-2015.12.01][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [_na_]]]; nested: ElasticsearchParseException[Failed to derive xcontent];}

Upvotes: 0

Views: 1208

Answers (2)

FuzzyAmi
FuzzyAmi

Reputation: 8157

Correctly building queries for Elasticsearch's DSL is a pain. its just so easy to get it wrong. For most usecases I just use either the query-builder in the Head plugin - or the SQL-to-ES plugin.

Both provide a simple UI for generating queries - and you can convert the result to json and just use that in your code.

This requires some work to install, but if you need to formulate lots of ES queries, it really pays off.

head plugin - does alot more than just building queries.

sql plugin

Upvotes: 0

moliware
moliware

Reputation: 10288

The q parameter seems to be wrong (the " character is in wrong position), try this:

http://localhost:9200/logstash-*/_search?pretty=true&q=path:"/logs/stats/container/application_1448508514184_0178/container_e06_1448508514184_0178_01_003568/stderr"&sort=@timestamp&size=100000

On the other hand,the term query is valid but it must be within query key, something like:

import requests
import json

esurl = "http://localhost:9200/logstash-*/_search"
r = requests.post(esurl,data=json.dumps({"query": {"term":{'path':'/logs/stats/container/application_1448508514184_0178/container_e06_1448508514184_0178_01_003568/stderr'}}}))
r.text

Upvotes: 2

Related Questions