CODEWITHSUNDEEP
elasticsearchsearchdiff
Luis Henrique
Luis Henrique

Reputation: 771

Query return the search difference on elasticsearch

How would the following query look:

Scenario:

I have two bases (base 1 and 2), with 1 column each, I would like to see the difference between them, that is, what exists in base 1 that does not exist in base 2, considering the fictitious names of the columns as hostname.

Example:

Selected value of Base1.Hostname is for Base2.Hostname?

YES → DO NOT RETURN
NO  → RETURN

I have this in python for the following function:

def diff(first, second):
        second = set (second)
        return [item for item in first if item not in second]

Example match equal:

GET /base1/_search
{
  "query": {
    "multi_match": {
      "query": "webserver",
      "fields": [
        "hostname"
      ],
      "type": "phrase"
    }
  }
}

I would like to migrate this architecture to elastic search in order to generate forecast in the future with the frequency of change of these search in the bases

Upvotes: 0

Views: 45

Answers (1)

Ashraful Islam
Ashraful Islam

Reputation: 12840

This could be done with aggregation.

  1. Collect all the hostname from base1 & base2 index
  2. For each hostname count occurrences in base2
  3. Keep only the buckets that have base2 count 0
GET base*/_search
{
  "size": 0,
  "aggs": {
    "all": {
      "composite": {
        "size": 10, 
        "sources": [
          {
            "host": {
              "terms": {
                "field": "hostname"
              }
            }
          }
        ]
      },
      "aggs": {
        "base2": {
          "filter": {
            "match": {
              "_index": "base2"
            }
          }
        },
        "index_count_bucket_filter": {
          "bucket_selector": {
            "buckets_path": {
              "base2_count": "base2._count"
            },
            "script": "params.base2_count == 0"
          }
        }
      }
    }
  }
}

By the way don't forget to use pagination to get rest of the result.

References :

  1. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html
  2. https://discuss.elastic.co/t/data-set-difference-between-fields-on-different-indexes/160015/4

Upvotes: 1

Related Questions