CODEWITHSUNDEEP
elasticsearchkibana
codeaprendiz
codeaprendiz

Reputation: 3195

Simple Example of ElasticSearch Index LifeCycle Policy in Kibana

Version of Elastic Search 7.10.2 and Xpack is enabled and the licence is Basic

Hot Phase of metricbeat policy

enter image description here

Delete Phase of metricbeat policy

enter image description here

WHY IS IT NOT GETTING APPLIED ?

enter image description here

metricbeat-7.10.2-2021.02.10-000001 index details

{
  "indices" : {
    "metricbeat-7.10.2-2021.02.10-000001" : {
      "index" : "metricbeat-7.10.2-2021.02.10-000001",
      "managed" : true,
      "policy" : "metricbeat",
      "lifecycle_date_millis" : 1612959479882,
      "age" : "8m",
      "phase" : "hot",
      "phase_time_millis" : 1612959480192,
      "action" : "rollover",
      "action_time_millis" : 1612959917863,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1612959917863,
      "phase_execution" : {
        "policy" : "metricbeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "5b",
              "max_age" : "5s",
              "max_docs" : 5
            }
          }
        },
        "version" : 2,
        "modified_date_in_millis" : 1612959551839
      }
    }
  }
}

Upvotes: 0

Views: 1831

Answers (2)

ylev
ylev

Reputation: 2569

The following step-by-step guide will assist to arrange Index Lifecycle Rollover Policy, Index Template, and common alias, indexing documents into the common alias and query by range

Step 1: Create an Index Lifecycle Policy

PUT _ilm/policy/test-index-policy1
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": 10000
          }
        }
      }
    }
  }
}

Step 2: Create an Index Template

PUT _index_template/test-index-template1
{
  "index_patterns": ["test-index-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "test-index-policy1",  // Link to the ILM policy1
      "index.lifecycle.rollover_alias": "test-index-alias"  // Alias to manage the rollover
    }
  }
}

Step 3: Create the Initial Index with an Alias

PUT test-index-000001
{
  "aliases": {
    "test-index-alias": {
      "is_write_index": true
    }
  }
}

Step 4: Insert (Bulk) Documents to the alias (python program example)

from elasticsearch import Elasticsearch, helpers
import sys

es = Elasticsearch("http://localhost:9200")   # Connect to Elasticsearch server

index_name = 'test-index-alias'  ## define Alias name

# Define a document template to be inserted to docs[] list
doc_tmplt = {
  "_id": 0,
  "_index": index_name,
  "subject":"TEST_4",
  "hostname": "TestMe",
  "myTimestamp": 0
}

docs = []
numOfDocs = 40000
startId = 150001
tm = 1719942000

# populate the document list docs[]
for x in range(startId, startId + numOfDocs): ## 0..numOfDocs-1
    cur_dict = dict(doc_tmplt) ## clone dict
    
    cur_dict['_id'] = x
    cur_dict['myTimestamp'] = tm
    cur_dict['hostname'] = f'{cur_dict['hostname']}_{x}'
    tm = tm + 1
    docs.append(cur_dict)

# Perform the bulk insert
res = helpers.bulk(es, docs)
sys.exit(0)

Summary

Following these steps will ensure that your Elasticsearch index rolls over to a new index after reaching 10,000 documents.

Terms

Index Lifecycle Policy: Defines when to rollover the index. Index Template : Applies the ILM policy and settings to indices matching a pattern.

Initial Index Creation: Creates the first index with the correct alias and settings.

Now, when you index documents into test-index-alias, Elasticsearch will automatically roll over to a new index (e.g., test-index-000002) once the document count in test-index-000001 reaches 100,000.

Queries:

To find 'myTimestamp' min, max values within the common alias:

POST test-index-alias/_search

POST test-index-000004/_search?pretty=true
{
  "size": 0,
  "aggs": {
    "document_count": {
      "value_count": {
        "field": "myTimestamp"
      }
    },
    "min_myTimestamp": {
      "min": {
        "field": "myTimestamp"
      }
    },
    "max_myTimestamp": {
      "max": {
        "field": "myTimestamp"
      }
    }
  }
}

To count 'myTimestamp' min, max values between range, in the common alias:

 POST test-index-alias/_search
{
  "size": 0,
  "query": {
    "range": {
      "myTimestamp": {
        "gte": 1719792000,
        "lte": 1720021999
      }
    }
  },
  "aggs": {
    "document_count": {
      "value_count": {
        "field": "myTimestamp"
      }
    },
    "min_myTimestamp": {
      "min": {
        "field": "myTimestamp"
      }
    },
    "max_myTimestamp": {
      "max": {
        "field": "myTimestamp"
      }
    }
  }
}

Upvotes: 2

Val
Val

Reputation: 217334

If a policy is modified AFTER an index has been created, it might not kick in as you expect.

ILM runs every 10 minutes by default, but that can be changed via the indices.lifecycle.poll_interval cluster setting.

Upvotes: 2

Related Questions