CODEWITHSUNDEEP
elasticsearchelasticsearch-curator
BlackHoleGalaxy
BlackHoleGalaxy

Reputation: 9662

Is it possible with Elastic Curator to delete indexes matching field value

We have on our elasticsearch several indexes. They come from FluentD pluging sendings logs fron our docker containers. We would like to delete old indexes not only older than specific amount of days based on index name but applying different delete rules depending on log fields.

Here is an example of log:

{
  "_index": "fluentd-2018.03.28",
  "_type": "fluentd",
  "_id": "o98123bcbd_kqpowkd",
  "_version": 1,
  "_score": null,
  "_source": {
    "container_id": "bbd72ec5e46921ab8896a05684a7672ef113a79e842285d932f",
    "container_name": "/redis-10981239d5",
    "source": "stdout",
    "log": "34:M 28 Mar 15:07:51.086 * 10 changes in 300 seconds. Saving...\r34:M 28 Mar 15:07:51.188 * Background saving terminated with success\r",
    "@timestamp": "2018-03-28T15:07:56.217739954+00:00",
    "@log_name": "docker.redis"
  },
  "fields": {
    "@timestamp": [
      "2018-03-28T15:07:56.217Z"
    ]
  }
}

In that case, we would like to delete all logs matching @log_name = docker.redis older than 7 days.

Is it possible to define a Curator action which deletes indexes filtered by such a field value?

We tried different filtering without any success. The only action we manage to perform successfully is based on index name:

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 30 days
    options:
      ignore_empty_list: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: fluentd-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30

Upvotes: 1

Views: 976

Answers (1)

Dario Balinzo
Dario Balinzo

Reputation: 691

Curator offer only an index level retention configuration. If you need a retention based on document level, you can try with a script that execute a delete by query.

Otherwise, using curator, you need to separate your data in different indexes for applying different retention.

Upvotes: 3

Related Questions