CODEWITHSUNDEEP
elasticsearchopensearch
AlertWar
AlertWar

Reputation: 11

AWS Elasticsearch Rollover Index with dates in the index name

I want to have index with date in the index name that automatically rolls over everyday. As for testing, I have ISM policy that rolls over everyhour

{
    "policy_id": "test",
    "description": "hot to warm every hour.",
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_doc_count": 1,
                        "min_index_age": "1h"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "warm",
                    "conditions": {
                        "min_index_age": "2h"
                    }
                }
            ]
        },
        {
            "name": "warm",
            "actions": [
                {
                    "retry": {
                        "count": 5,
                        "backoff": "exponential",
                        "delay": "1m"
                    },
                    "warm_migration": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "test-*"
            ],
            "priority": 100,
            "last_updated_time": 1640874381088
        }
    ]
}

And Index Template with rollover alias

PUT /_template/test_template
{ 
  "index_patterns" : ["test-*"],
  "settings" : {
    "index" : {
      "opendistro.index_state_management.rollover_alias": "test_alias",
      "refresh_interval" : "1s",
      "number_of_shards" : "1",
      "number_of_replicas" : "1"
    }
  }
}

Then I created initial rollover index

# PUT <test-{now/h{YYYYMMddHH}}>
PUT %3Ckbtest-ai-detector-%7Bnow%2Fh%7BYYYYMMdd%7D%7D%3E
{
  "aliases": {
    "test_alias": {
      "is_write_index": true
    }
  }
}

which created test-2021123016. But whenever ISM tries to roll over, I get following error

{
    "cause": "For input string: \"{now/h{YYYYMMddHH}}\"",
    "message": "Failed to rollover index [index=test-2021123016]"
}

Is it not able to use date math expression when it comes to ISM policy?

Upvotes: 1

Views: 2875

Answers (2)

Sagar Vaghela
Sagar Vaghela

Reputation: 1263

If you need only date to add in your rolled over index then you can use below format.

PUT /%3Ckbtest-ai-detector-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "test_alias":{
      "is_write_index": true 
    }
  }
}

This will add the date on the new roll over index name automatically. For example.

kbtest-ai-detector-2022.02.10-000001
kbtest-ai-detector-2022.02.11-000002
kbtest-ai-detector-2022.02.12-000003
kbtest-ai-detector-2022.02.13-000004

Upvotes: 0

ilvar
ilvar

Reputation: 5841

You don't need ISM for this case, just construct index name in your app when indexing (using current time like test-2021-12-30; use index template to control settings and mappings) and use wildcard index for searching (test-*?).

Upvotes: 1

Related Questions