Range query in elasticsearch does not work properly

Question

I have an index that contains objects eventvalue-eventtime. I want to write a query that will return aggregated event count based on eventvalue for the last 30 seconds. Also, I need empty buckets if for a given seconds there was no events - I need to display this data on a graph.

So I wrote the following query:

{
    "query" : {
        "bool" : {
            "must" : [
                {
                    "range" : { 
                        "eventtime" : {
                            "gte" : "now-30s/s",
                            "lte" : "now/s",
                            "format" : "yyyy-MM-dd HH:mm:ss",
                            "time_zone": "+03:00"
                        }
                    }
                },
                {
                    "range" : { 
                        "eventvalue" : {
                            "lte" : 3
                        }
                    }
                }
            ]
        }
    },
    "aggs": {
        "values_agg": {
            "terms": {
                "field": "eventvalue",
                "min_doc_count" : 0,
                "order": {
                    "_term": "asc"
                }
            },
            "aggs": {
                "events_over_time" : {
                    "date_histogram" : {
                        "field" : "eventtime",
                        "interval" : "1s",
                        "min_doc_count" : 0,
                        "extended_bounds" : {
                            "min" : "now-30s/s",
                            "max" : "now/s"
                        },
                        "format" : "yyyy-MM-dd HH:mm:ss",
                        "time_zone": "+03:00"
                    }
                }
            }
        }
    }
}

This query is not working properly and I don't know why. Specifically, the first "range" query gives me desired interval (if I remove it I'm getting values from all time). But the second "range" query seems to have no effect. Eventvalue can be anywhere from 1 to 10 and the desired effect is that I will have three buckets for eventvalues 1-3. However, I get all 10 buckets with all events.

How can I fix this query so it still returns empty buckets but only for selected evenvalues?

Answer 1

I believe you need to remove the "min_doc_count": 0 from your terms aggregation. To achieve the empty buckets you're aiming for, you need only use min_doc_count in the date_histogram aggregation.

Per the documentation for the terms aggregation:

Setting min_doc_count=0 will also return buckets for terms that didn’t match any hit.

This explains why you are seeing buckets for eventvalues that are greater than 3. They were filtered out by the query, but brought back in by the terms aggregation.

UPDATE

Since there is a possibility that the eventvalues may not exist anywhere in the 30sec time slice, the other approach I would recommend is to manually specify the discrete values you want to use as buckets using a filters aggregation. See the documentation here.

Try using this for your aggregations:

"aggs": {
  "values_agg": {
    "filters": {
      "filters": {
        "1": { "term": { "eventvalue": 1 }},
        "2": { "term": { "eventvalue": 2 }},
        "3": { "term": { "eventvalue": 3 }}
      }
    },
    "aggs": {
      "events_over_time" : {
        "date_histogram" : {
          "field" : "eventtime",
          "interval" : "1s",
          "min_doc_count" : 0,
          "extended_bounds" : {
            "min" : "now-30s/s",
            "max" : "now/s"
          },
          "format" : "yyyy-MM-dd HH:mm:ss",
          "time_zone": "+03:00"
        }
      }
    }
  }
}