Elasticsearch: Scripted Metric Aggregation returns weird grouping

Question

I'm getting an unexpected result and I'm a bit lost.

I added these docs:

POST es_test/_doc/_bulk?pretty
{ "index": {}}
{ "firstname": "John", "lastname": "Doe", "age": 22, "birthdate": "1980-01-20T12:30:00Z" }
{ "index": {}}
{ "firstname": "May", "lastname": "Greenwood", "age": 19, "birthdate": "1980-01-20T12:30:00Z" }
{ "index": {}}
{ "firstname": "Marry", "lastname": "Hilake", "age": 32, "birthdate": "1970-01-20T12:30:00Z" }
{ "index": {}}
{ "firstname": "Mister", "lastname": "X", "age": 20, "birthdate": "1990-11-23T12:30:00Z" }

It's all good when I request them like this:

GET es_test/_doc/_search

The problem occurs when I add a painless script:

GET es_test/_doc/_search    
{
  "size": 0,
  "aggs": {
    "user": {
      "scripted_metric": {
        "init_script" : "params._agg.transactions = []",
        "map_script" : "params._agg.transactions.add(params._source)"
      }
    }
  }
}

The output of it looks like this:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 4,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "user": {
      "value": [
        {
          "transactions": [
            {
              "firstname": "John",
              "birthdate": "1980-01-20T12:30:00Z",
              "age": 22,
              "lastname": "Doe"
            },
            {
              "firstname": "John",
              "birthdate": "1980-01-20T12:30:00Z",
              "age": 22,
              "lastname": "Doe"
            }
          ]
        },
        {
          "transactions": []
        },
        {
          "transactions": [
            {
              "firstname": "May",
              "birthdate": "1980-01-20T12:30:00Z",
              "age": 19,
              "lastname": "Greenwood"
            }
          ]
        },
        {
          "transactions": []
        },
        {
          "transactions": [
            {
              "firstname": "Marry",
              "birthdate": "1970-01-20T12:30:00Z",
              "age": 32,
              "lastname": "Hilake"
            }
          ]
        }
      ]
    }
  }
}

The first transactions array includes two times John, the second one is empty, then comes May, empty again and finally Marry. I have no idea why it's grouped so weird.

Desired output would be one array that includes all users (John, May, Marry, Mister).

I appreciate your help, thank you! Game

Answer 1

You'll have to use a reduce_script for that. This works

GET es_test/_doc/_search
{
  "size": 0,
  "aggs": {
    "user": {
      "scripted_metric": {
        "init_script": "params._agg.transactions = []",
        "map_script": "params._agg.transactions.add(params._source)",
        "reduce_script": """
                ArrayList transactions = []; 
                for (a in params._aggs) { 
                  transactions.addAll(a.transactions) 
                } 
                return transactions
"""
      }
    }
  }
}

Result

{
  "took": 12,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 4,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "user": {
      "value": [
        {
          "firstname": "John",
          "birthdate": "1980-01-20T12:30:00Z",
          "age": 22,
          "lastname": "Doe"
        },
        {
          "firstname": "Marry",
          "birthdate": "1970-01-20T12:30:00Z",
          "age": 32,
          "lastname": "Hilake"
        },
        {
          "firstname": "Mister",
          "birthdate": "1990-11-23T12:30:00Z",
          "age": 20,
          "lastname": "X"
        },
        {
          "firstname": "Mister",
          "birthdate": "1990-11-23T12:30:00Z",
          "age": 20,
          "lastname": "X"
        }
      ]
    }
  }
}

---

Update, the below works, which is the same aggregation, but it only uses part of the source, which is very odd.

GET es_test/_search
{
  "size": 0,
  "query": {
    "match_all": {}
  },
  "aggs": {
    "user": {
      "scripted_metric": {
        "params": {
          "_agg": {}
        },
        "init_script": "params._agg.transactions = []",
        "map_script": "params._agg.transactions.add(params._source.firstname + ' ' + params._source.lastname)",
        "reduce_script": """
                ArrayList transactions = []; 
                for (a in params._aggs) { 
                  transactions.addAll(a.transactions) 
                } 
                return transactions
"""
      }
    }
  }
}

Result

{
  "took": 18,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 4,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "user": {
      "value": [
        "John Doe",
        "May Greenwood",
        "Marry Hilake",
        "Mister X"
      ]
    }
  }
}