CODEWITHSUNDEEP
elasticsearch
Agung Darmanto
Agung Darmanto

Reputation: 419

Subtract numeric fields between two documents with different timestamp

Lets say I have these data samples:

{
    "date": "2019-06-16",
    "rank": 150
    "name": "doc 1"
}

{
    "date": "2019-07-16",
    "rank": 100
    "name": "doc 1"
}

{
    "date": "2019-06-16",
    "rank": 50
    "name": "doc 2"
}

{
    "date": "2019-07-16",
    "rank": 80
    "name": "doc 2"
}

The expected result is by subtracting the rank field from two same name of docs with different date (old date - new date):

{
    "name": "doc 1",
    "diff_rank": 50
}

{
    "name": "doc 2",
    "diff_rank": -30
}

And sort by diff_rank if possible, otherwise I will just sort manually after getting the result.

What I have tried is by using date_histogram and serial_diff but some results are missing the diff_rank value in somehow which I am sure the data exist:

{
   "aggs" : {
        "group_by_name": {
            "terms": {
                "field": "name"
            },
            "aggs": {
                "days": {
                    "date_histogram": {
                        "field": "date",
                        "interval": "day"
                     },
                    "aggs": {
                        "the_rank": {
                            "sum": {
                                "field": "rank"
                            }
                        },
                        "diff_rank": {
                           "serial_diff": {
                              "buckets_path": "the_rank",
                              "lag" : 30 // 1 month or 30 days in this case
                           }
                        }
                    }
                }
            }
        }
    }
}

The help will be much appreciated to solve my issue above!

Upvotes: 4

Views: 1386

Answers (1)

Agung Darmanto
Agung Darmanto

Reputation: 419

Finally, I found a method from official doc using Filter, Bucket Script aggregation and Bucket Sort to sort the result. Here is the final snippet code:

{
    "size": 0,
    "aggs" : {
        "group_by_name": {
            "terms": {
                "field": "name",
                "size": 50,
                "shard_size": 10000
            },
            "aggs": {
                "last_month_rank": {
                    "filter": {
                        "term": {"date": "2019-06-17"}
                     },
                    "aggs": {
                        "rank": {
                            "sum": {
                                "field": "rank"
                            }
                        }
                    }
                },
                "latest_rank": {
                    "filter": {
                        "term": {"date": "2019-07-17"}
                     },
                    "aggs": {
                        "rank": {
                            "sum": {
                                "field": "rank"
                            }
                        }
                    }
                },
                "diff_rank": {
                    "bucket_script": {
                        "buckets_path": {
                          "lastMonthRank": "last_month_rank>rank",
                          "latestRank": "latest_rank>rank"
                        },
                        "script": "params.lastMonthRank - params.latestRank"
                    }
                },
                "rank_bucket_sort": {
                    "bucket_sort": {
                        "sort": [
                            {"diff_rank": {"order": "desc"}}
                        ],
                        "size": 50
                    }
                }
            }
        }
    }
}

Upvotes: 1

Related Questions