Index Lifecycle policy for deleting old indices

Question

I've read the following tutorial:

https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html

After following its stages for several times, I'm still stuck getting the same errors. I would like to automatically delete old indices, that are at a certain age.

I have several instances on AWS, each instance is writing to its own index on elastic-search e.g.: index name -

filebeat-log-centralization-ds-test-2020.08.09

which is mapped to an index template

{
  "filebeat-template" : {
    "order" : 0,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "logs-deletion-policy",
          "rollover_alias" : "filebeat"
        },
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : {
      "_meta" : { },
      "_source" : { },
      "properties" : { /*** some mappings ***/
    "aliases" : { }
  }
}

ILM policy :

PUT _ilm/policy/logs-deletion-policy
{
  "policy": {
    "phases": {
      "hot": {                      
        "actions": {
          "rollover": {
            "max_size": "50GB",     
            "max_age": "1m"
          }
        }
      },
      "delete": {
        "min_age": "0d",           
        "actions": {
          "delete": {}              
        }
      }
    }
  }
}

I've bootstrapped the rollover alias index :

PUT filebeat-000001
{
  "aliases": {
    "filebeat": {
      "is_write_index": true
    }
  }
}

The filebeat-0000N index (rollover alias index i've just created) is rolling over and being deleted fine, however "filebeat-log-centralization-ds-test-2020.08.09" is showing an error :

illegal_argument_exception: index.lifecycle.rollover_alias [filebeat] 
does not point to index [filebeat-log-centralization-ds-test-2020.08.09]

I Have no idea what am I doing wrong, and tried to repeat the tutorial with no success. What am I missing?

EDIT:

I've tried adding the filebeat alias to the index template.

GET /filebeat-log-centralization-ds-test-2020.08.09/_alias/
{
  "filebeat-log-centralization-ds-test-2020.08.09" : {
    "aliases" : {
      "filebeat" : { }
    }
  }
}

which results in an error for the filebeat-log-centralization-ds-test-2020.08.09 index

illegal_argument_exception: Rollover alias [filebeat] can point to multiple indices, found duplicated alias [[filebeat]] in index template [filebeat-template]

Tomer.Ov · Accepted Answer

I could not fix this issue, but did solve it using curator, since my main goal was to delete old indices.

Create a file called config.yml

client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: username:password
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:  /var/log/curator/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Curator needs an action file. Create a file called delete_indices.yml

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices with age greater than 4 days(based on index name), for filebeat-*
      prefixed indices.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 4

Above config deletes all indices with age greater than 4 days

Now you can run it using the following command -

curator --config config.yml delete_indices.yml

Since my configuration required automatically deleting old indices, I'm running it using crontab.

Run

crontab -e

This will open your personal crontab (cron configuration file).

Add the following code-

* * * * * /usr/local/bin/curator --config {config.yml location} {delete_indices.yml location} >/dev/null 2>&1

Index Lifecycle policy for deleting old indices

Answers (2)

Related Questions