CODEWITHSUNDEEP
elasticsearchelasticsearch-dsl
Sandeep Kanabar
Sandeep Kanabar

Reputation: 1302

Elasticsearch finding which indices contain hits when making use of an alias in query

I've 1000s of day-wise (time-based) indices in my cluster. Is there a way to monitor or find out at the ES Server side which indices contain hits when a query makes use of an alias . E.g. consider the below query that my customer fires:

GET index_alias/_search?q=Product_Serial_Num:ABCD

The alias index_alias is mapped to all the day-wise indices in the ES 6.8.6 cluster and they total 1200. The reason being the query is used to fetch historical records for that Product_Serial_Num. Thus it is not possible to know beforehand which day-wise index would have data. That's the reason the index_alias is mapped to all indices. When the customer fires the above query, I would like to monitor this at the ES Server side and find out which indices actually return the data when the query hits them. Is there a way to determine this without modifying anything at the customer end?

The purpose of this is to figure out the "which indices" are being hit the most, like last-n-days (30, 60, 90) and then apply hot-warm architecture accordingly like having last 60 days indices on hot nodes and rest on warm nodes.

Or if there are better ways to determine, please do share.

Thanks.

Upvotes: 0

Views: 1625

Answers (2)

Jaycreation
Jaycreation

Reputation: 2089

To complete @opster ES Ninja Answer. You can have a part of information you need with _stats api

In the search part, for each index, you will be able to know wish index is the most used

GET my_alias/_stats
"indices" : {
    "metricbeat-7.3.1-2019.12.05-000004" : {
      "uuid" : "2Jgl5tAfRBCUC8UjhzJMXw",
      "primaries" : {
        "docs" : {
          "count" : 4536039,
          "deleted" : 0
        },
        "store" : {
          "size_in_bytes" : 1314781408,
          "reserved_in_bytes" : 0
        },
        "indexing" : {
          "index_total" : 0,
          "index_time_in_millis" : 0,
          "index_current" : 0,
          "index_failed" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0,
          "noop_update_total" : 0,
          "is_throttled" : false,
          "throttle_time_in_millis" : 0
        },
        "get" : {
          "total" : 0,
          "time_in_millis" : 0,
          "exists_total" : 0,
          "exists_time_in_millis" : 0,
          "missing_total" : 0,
          "missing_time_in_millis" : 0,
          "current" : 0
        },
        "search" : {
          "open_contexts" : 0,
          "query_total" : 3,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0,
          "scroll_total" : 0,
          "scroll_time_in_millis" : 0,
          "scroll_current" : 0,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        },
        ...

Upvotes: 1

Nishant
Nishant

Reputation: 7864

You should be already getting the information about the index in every hit metadata. Look at the sample response below:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test_index_01",
        "_type" : "_doc",
        "_id" : "doc_id_1",
        "_score" : 1.0
      },
      {
        "_index" : "test_index_01",
        "_type" : "_doc",
        "_id" : "doc_id_5",
        "_score" : 1.0
      }
    ]
  }
}

In the above response you can see that each hit contains meta fields and one such field is _index. The value of this field is actual name of index (not alias) where the document is found. You can use this information as per your needs.

The other way to find which index the alias is currently pointing to can be done by using GET alias api.

GET _alias/test_alias

Sample response:

{
  "test_index_01" : {
    "aliases" : {
      "test_alias" : { }
    }
  }
}

The above response provides the info about the test_alias.

Upvotes: 2

Related Questions