Elasticsearch aggregation of deep nested type

Question

Previously I have asked this question.

The example document there was a simplified document. That was good for me to understand the differences in aggregation over non-nested type versus nested type. However, the simplification was hiding further complexity and so I have to expand on the question here.

So my actual documents are closer to the following:

"_source": {
    "keyword": "my keyword",
    "response": [
        {
            "results": [
                {
                    "items": [
                        {
                            "prop": [
                                {
                                    "item_property_1": ["A"],
                                }
                            ]
                            ( ... other properties )
                        },
                        {
                            "prop": [
                                {
                                    "item_property_1": ["B"],
                                }
                            ]
                            ( ... other properties )
                        },
                        ( ... other items )
                    ]
                }
            ],
            ( ... other properties )
        }
    ]
}

So I kept the crucial properties keyword, items, and item_property_1, but hid lots of other things that complicate the situation. First, notice that compared to the referenced question there is lots of extra nesting: between the root and "items", and between "items" and "item_property_1". Additionally, notice also that the properties response and results are both arrays with a single element. It's weird, but that's how it is :-)

Now, the reason why this question is different from the one cited above is that I tried the accepted answer (which does work for the example there), and it doesn't work here. That is, if I use a mapping with:

"items": {
    "type":"nested",
    "properties": {
        "prop": {
            "properties": {
                "item_property_1": {
                    "type": "string",
                    "index": "not_analyzed"
                },
            }
        }
    }
}

then the aggregation doesn't work. It returns zero hits.

I will edit later and provider a ready to use sample bulk insert.

EDIT: Alright, below I show three queries which are respectively: mapping, bulk insert and aggregation (with zero hits)

Mapping (with "type":"nested" as indicated in the previous answered question)

PUT /test2/_mapping/test3
{
    "test3": {
        "properties": {
            "keyword": {
                "type": "string",
                "index": "not_analyzed"
            },
            "response": {
                "properties": {
                    "results": {
                        "properties": {
                            "items": {
                                "type": "nested",
                                "properties": {
                                    "prop": {
                                        "properties": {
                                            "item_property_1": {
                                                "type": "string",
                                                "index": "not_analyzed"
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

Bulk put:

PUT /test2/test3/_bulk
{ "index": {}}
{    "keyword": "my keyword",    "response": [        {            "results": [                {                    "items": [                        {                            "prop": [                                {"item_property_1": ["A"]}                            ]                        },                        {                            "prop": [                                {"item_property_1": ["B"]}                            ]                        },                        {                            "prop": [                                {"item_property_1": ["A"]}                            ]                        }                    ]                }            ]        }    ]}
{ "index": {}}
{    "keyword": "different keyword",    "response": [        {            "results": [                {                    "items": [                        {                            "prop": [                                {"item_property_1": ["A"]}                            ]                        },                        {                            "prop": [                                {"item_property_1": ["C"]}                            ]                        }                    ]                }            ]        }    ]}

Aggregation (zero hits):

POST /test2/test3/_search
{
    "size":0,
    "aggregations": {
        "item_property_1_count": {
            "terms":{
                "field":"item_property_1"
            }
        }
    }
}

Answer 1

It's not really different from the previous answer. All you need is to modify the field names a little bit to take into account the additional nesting. Other than that, nothing needs to change in the mapping. Note that this query works without mapping changes only because response and results are both arrays with a single element, if it wasn't the case, it would be more involved and would require mapping changes and a different query.

The query now looks like this:

{
  "size": 0,
  "aggregations": {
    "by_keyword": {
      "terms": {
        "field": "keyword"
      },
      "aggs": {
        "prop_1_count": {
          "nested": {
            "path": "response.results.items"
          },
          "aggs": {
            "prop_1": {
              "terms": {
                "field": "response.results.items.prop.item_property_1"
              }
            }
          }
        }
      }
    }
  }
}

And the results:

{
  ...
  "aggregations" : {
    "by_keyword" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ {
        "key" : "different keyword",
        "doc_count" : 1,
        "prop_1_count" : {
          "doc_count" : 2,
          "prop_1" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "A",
              "doc_count" : 1
            }, {
              "key" : "C",
              "doc_count" : 1
            } ]
          }
        }
      }, {
        "key" : "my keyword",
        "doc_count" : 1,
        "prop_1_count" : {
          "doc_count" : 3,
          "prop_1" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "A",
              "doc_count" : 2
            }, {
              "key" : "B",
              "doc_count" : 1
            } ]
          }
        }
      } ]
    }
  }
}