Reputation: 2717
Query on number field matching string fields
I have an ELK stack setup. When I am performing a query on number fields then it is also matching against string fields. For example, I am sending Load Balancer logs to ELK and if I perform backend_processing_time:>5 on that then it is matching against backend_processing_time with value 0.001 too.
On kibana interface, it is showing that the query is matching string in the request message. I am not able to understand how a query against a number field is matching against a string.
In the dev tools section on kibana i tried to run the same query
GET _search
{
"query": {
"range" : {
"backend_processing_time" : {
"gte" : 50000000000
}
}
}
}
Even with so much backend_processing_time i am getting results. I am not able to understand why this is happening.
I searched on other fields also which are of number type and found that all the queries done on number field are getting matched with string type fields.
I am providing a sample search result which i get for backend_processing_time:>500000000 query. It can be seen in this result that backend_processing_time field is so small but still getting a hit.
{
"_index": "logstash-2017.05.10",
"_type": "prod-quizelb-logs",
"_id": "AVvzYRgL49GPTZAKoDer",
"_score": null,
"_source": {
"backendport": 80,
"received_bytes": 0,
"request": "http://en.meaww.com:80/locales/en.json",
"backend_response": 200,
"verb": "GET",
"message": "2017-05-10T17:19:52.881044Z Prod-ELB 172.68.144.71:34803 10.1.91.253:80 0.000075 0.000606 0.000019 200 200 0 1881 \"GET http://en.meaww.com:80/locales/en.json HTTP/1.1\" \"Mozilla/5.0 (Linux; Android 6.0.1; SM-C900F Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/58.0.3029.83 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/122.0.0.17.71;]\" - -\n",
"type": "prod-quizelb-logs",
"clientport": 34803,
"request_processing_time": 0.000075,
"urihost": "en.meaww.com:80",
"response_processing_time": 0.000019,
"path": "/locales/en.json",
"@timestamp": "2017-05-10T17:21:18.280Z",
"port": "80",
"response": 200,
"bytes": 1881,
"clientip": "172.68.144.71",
"proto": "http",
"@version": "1",
"elb": "Prod-ELB",
"httpversion": "1.1",
"backendip": "10.1.91.253",
"backend_processing_time": 0.000606,
"timestamp": "2017-05-10T17:19:52.881044Z"
},
"fields": {
"@timestamp": [
1494436878280
],
"timestamp": [
1494436792881
]
},
"highlight": {
"backend_processing_time.keyword": [
"@[email protected]@/kibana-highlighted-field@"
],
"request": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@://@[email protected]@/kibana-highlighted-field@:@kibana-highlighted-field@80@/kibana-highlighted-field@/@kibana-highlighted-field@locales@/kibana-highlighted-field@/@[email protected]@/kibana-highlighted-field@"
],
"elb.keyword": [
"@kibana-highlighted-field@Prod-ELB@/kibana-highlighted-field@"
],
"urihost.keyword": [
"@[email protected]:80@/kibana-highlighted-field@"
],
"verb": [
"@kibana-highlighted-field@GET@/kibana-highlighted-field@"
],
"request.keyword": [
"@kibana-highlighted-field@http://en.meaww.com:80/locales/en.json@/kibana-highlighted-field@"
],
"type": [
"@kibana-highlighted-field@prod@/kibana-highlighted-field@-@kibana-highlighted-field@quizelb@/kibana-highlighted-field@-@kibana-highlighted-field@logs@/kibana-highlighted-field@"
],
"message": [
"2017-05-10T17:19:@[email protected]@/kibana-highlighted-field@ @kibana-highlighted-field@Prod@/kibana-highlighted-field@-@kibana-highlighted-field@ELB@/kibana-highlighted-field@ 172.68.144.71:34803 10.1.91.253:@kibana-highlighted-field@80@/kibana-highlighted-field@ 0.000075 0.000606 0.000019 200 200 0 1881 \"@kibana-highlighted-field@GET@/kibana-highlighted-field@ @kibana-highlighted-field@http@/kibana-highlighted-field@://@[email protected]@/kibana-highlighted-field@:@kibana-highlighted-field@80@/kibana-highlighted-field@/@kibana-highlighted-field@locales@/kibana-highlighted-field@/@[email protected]@/kibana-highlighted-field@ @kibana-highlighted-field@HTTP@/kibana-highlighted-field@/1.1\" \"@kibana-highlighted-field@Mozilla@/kibana-highlighted-field@/5.0 (@kibana-highlighted-field@Linux@/kibana-highlighted-field@; @kibana-highlighted-field@Android@/kibana-highlighted-field@ @[email protected]@/kibana-highlighted-field@; @kibana-highlighted-field@SM@/kibana-highlighted-field@-@kibana-highlighted-field@C900F@/kibana-highlighted-field@ @kibana-highlighted-field@Build@/kibana-highlighted-field@/@kibana-highlighted-field@MMB29M@/kibana-highlighted-field@; @kibana-highlighted-field@wv@/kibana-highlighted-field@) @kibana-highlighted-field@AppleWebKit@/kibana-highlighted-field@/@[email protected]@/kibana-highlighted-field@ (@kibana-highlighted-field@KHTML@/kibana-highlighted-field@, @kibana-highlighted-field@like@/kibana-highlighted-field@ @kibana-highlighted-field@Gecko@/kibana-highlighted-field@) @kibana-highlighted-field@Version@/kibana-highlighted-field@/4.0 @kibana-highlighted-field@Chrome@/kibana-highlighted-field@/@[email protected]@/kibana-highlighted-field@ @kibana-highlighted-field@Mobile@/kibana-highlighted-field@ @kibana-highlighted-field@Safari@/kibana-highlighted-field@/@[email protected]@/kibana-highlighted-field@ [@kibana-highlighted-field@FB_IAB@/kibana-highlighted-field@/@kibana-highlighted-field@FB4A@/kibana-highlighted-field@;@kibana-highlighted-field@FBAV@/kibana-highlighted-field@/122.0.0.17.71;]\" - -\n"
],
"urihost": [
"@[email protected]@/kibana-highlighted-field@:@kibana-highlighted-field@80@/kibana-highlighted-field@"
],
"path": [
"/@kibana-highlighted-field@locales@/kibana-highlighted-field@/@[email protected]@/kibana-highlighted-field@"
],
"verb.keyword": [
"@kibana-highlighted-field@GET@/kibana-highlighted-field@"
],
"proto.keyword": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@"
],
"port": [
"@kibana-highlighted-field@80@/kibana-highlighted-field@"
],
"type.keyword": [
"@kibana-highlighted-field@prod-quizelb-logs@/kibana-highlighted-field@"
],
"proto": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@"
],
"elb": [
"@kibana-highlighted-field@Prod@/kibana-highlighted-field@-@kibana-highlighted-field@ELB@/kibana-highlighted-field@"
],
"backend_processing_time": [
"@[email protected]@/kibana-highlighted-field@-4"
],
"port.keyword": [
"@kibana-highlighted-field@80@/kibana-highlighted-field@"
]
},
"sort": [
1494436878280
]
}
EDIT
I got the mapping by running GET /logstash-2017.05.11/_mapping/prod-quizelb-logs query in kibana console.
The mapping which I am getting for backend_processing_time is showing this
"backend_processing_time": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword"
}
}
}
So it seems that this field is of text type thus causing this error to happen.
Now I have another confusion i.e. kibana is showing this as number but elasticsearch is showing this of type text. Also, this is getting mapped dynamically as i never created the mapping on my own. I think that they are getting created by logstash at the time grok filter is applied.
Upvotes: 1
Views: 216
Answers (2)
Reputation: 52368
You need to take control of the mapping of those index(indices) so that your field will actually be a number. Otherwise, you will not be sure what kind of field type you'll have there. So, basically you need something like this, either in an index template, or a static mapping all the way:
"backend_processing_time": {
"type": "integer"
}
Upvotes: 1
Related Questions
- Elastic Search Json field query with number key
- Kibana Query Language - find numbers in a field
- How does one query an exact text-field and filter terms on another number-field in an ElasticSearch Index?
- How can I find all documents in elasticsearch that contain a number in a certain field?
- ElasticSearch: Query number field to use like clause as we do for string field
- Elasticsearch query on string representation of number
- How to make a field in Kibana numeric (from String)
- elasticsearch: extract number from a field
- How to perform full text search on number fields in ElasticSearch?
- Searching a number in a string field with query_string on Elasticsearch