CODEWITHSUNDEEP
elasticsearchkibanaelastic-stackelk
olg32
olg32

Reputation: 345

Problem with Elasticsearch Index Lifecycle Policy that doesn't rollover

In order to evaluate its potential to help on our daily operations, I have deployed Elastic Search and Kibana (7.7.1 with BASIC license) and created an index template for Ntopng (our monitoring platform).

Since indexes keep growing, I want to delete Ntopng indexes older than 20 days or so, therefore I have created a life cycle policy called ntopng where the time-stamped index should rollover after 1 day (for testing purposes) and then will be deleted after 2 days of the rollover:

enter image description here

Next I picked a time-stamped index created that day and applied the lifecycle policy to it:

enter image description here

Before that, I had to create an alias for that Index, so I did it manually:

POST /_aliases
{
  "actions" : [
    { "add" : { "index" : "ntopng-2020.09.09", "alias" : "ntopng_Alias" } }
  ]
}

All looked good after that ( I guess) as no errors or alarms were displayed:

"indices" : {
    "ntopng-2020.09.09" : {
      "index" : "ntopng-2020.09.09",
      "managed" : true,
      "policy" : "ntopng",
      "lifecycle_date_millis" : 1599609600433,
      "age" : "20.14h",
      "phase" : "hot",
      "phase_time_millis" : 1599681721821,
      "action" : "rollover",
      "action_time_millis" : 1599680521920,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1599681721821,
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 1,
      "phase_execution" : {
        "policy" : "ntopng",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 4,
        "modified_date_in_millis" : 1599509572867
      }
    }

My expectation was that in the next day the policy would be automatically rolled over to the next index (ntopng-2020.10.10) so that the initial index would be eventually deleted the next two days.

Instead, I got the following errors:

GET ntopng-*/_ilm/explain
{
  "indices" : {
    "ntopng-2020.09.09" : {
      "index" : "ntopng-2020.09.09",
      "managed" : true,
      "policy" : "ntopng",
      "lifecycle_date_millis" : 1599609600433,
      "age" : "1.94d",
      "phase" : "hot",
      "phase_time_millis" : 1599776521822,
      "action" : "rollover",
      "action_time_millis" : 1599680521920,
      "step" : "ERROR",
      "step_time_millis" : 1599777121822,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 80,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : """index name [ntopng-2020.09.09] does not match pattern '^.*-\d+$'""",
        "stack_trace" : """java.lang.IllegalArgumentException: index name [ntopng-2020.09.09] does not match pattern '^.*-\d+$'
    at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.generateRolloverIndexName(TransportRolloverAction.java:241)
    at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:133)
    at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:73)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:170)
    at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:170)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:133)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:110)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399)
    at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:92)
    at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:155)
    at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:51)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1234)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1736)
    at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:127)
    at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)
"""
      },
      "phase_execution" : {
        "policy" : "ntopng",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 4,
        "modified_date_in_millis" : 1599509572867
      }
    }


  "ntopng-2020.09.10" : {
      "index" : "ntopng-2020.09.10",
      "managed" : true,
      "policy" : "ntopng",
      "lifecycle_date_millis" : 1599696000991,
      "age" : "22.57h",
      "phase" : "hot",
      "phase_time_millis" : 1599776521844,
      "action" : "rollover",
      "action_time_millis" : 1599696122033,
      "step" : "ERROR",
      "step_time_millis" : 1599777121839,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 67,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index.lifecycle.rollover_alias [ntopng_Alias] does not point to index [ntopng-2020.09.10]",
        "stack_trace" : """java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [ntopng_Alias] does not point to index [ntopng-2020.09.10]
    at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104)
    at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)
"""
      },
      "phase_execution" : {
        "policy" : "ntopng",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        }

The first index error reads "index name [ntopng-2020.09.09] does not match pattern '^.*-\d+$"

while second one displays: ""index.lifecycle.rollover_alias [ntopng_Alias] does not point to index [ntopng-2020.09.10]"

Please note that I'm learning the basics on ES Index management, so I'd appreciate any clue on what the problem might be.

Upvotes: 3

Views: 14350

Answers (2)

JehanB
JehanB

Reputation: 26

As per the kibana regex you can date and time pattern as well, instead of the 2020.01.01 use 2020-01-01

This should work as well. You can check the regex here : https://regex101.com/r/VclptX/1 enter image description here

Upvotes: 1

olg32
olg32

Reputation: 345

OK, I just found that the index name must end with a numeric pattern like 0001 and not 2020.09.09 So I may need to find an alternative way to make it work.

Upvotes: 4

Related Questions