CODEWITHSUNDEEP
elasticsearchlogstash
Ricky
Ricky

Reputation: 324

How to use Elasticsearch rollover API where alias is updated to point to the latest created index?

I have a Logstash pipeline that runs on a daily schedule and pushes data to Elasticsearch using agent_index_%{+YYYY_MM_dd} index signature. So every day I would get a new index created eg: agent_index_2020_05_05 for 5th May 2020 and agent_index_2020_05_06 for 6th May 2020.

For this agent index alias is updated using it's mapping, which works fine, but my requirement is that the agent alias should always point to only 1 index that is the latest index.

mapping 

PUT /_template/agent_template
{
  "order": 0,
  "index_patterns": [
    "agent_index_*"
  ],
  "aliases": {
    "agent": {}
  }
}

I looked into this, but looks like the index name must end in incremental numbers.

POST /agent/_rollover?dry_run
{
  "conditions": {
    "max_age": "1d"
  }
}

Error

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "index name [agent_index_2020_05_06] does not match pattern '^.*-\\d+$'"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "index name [agent_index_2020_05_06] does not match pattern '^.*-\\d+$'"
  },
  "status": 400
}

Is there a way to use elasticsearch rollover API to only allow the latest created index to be updated in the alias, that means at a given point alias points to only one index which is the latest index?

Note: Using Elasticsearh v6.2.4

Update

My Logstash configuration

input {
    jdbc {
        jdbc_driver_library => "ojdbc7.jar"
        jdbc_driver_class    => "Java::oracle.jdbc.OracleDriver"
        jdbc_connection_string => "jdbc:oracle:thin:@EDM:1521/aba"
        jdbc_user => "read"
        jdbc_password => "read"
        schedule => "50 6 * * *"
        statement_filepath =>"agent.sql"
       }
}
output {
    elasticsearch {
        hosts => "localhost:17002"
        index => "agent_index_%{+YYYY_MM_dd}"
        document_type => "agent"
    }
}

Update

Using monitoring API to get details as suggested by @Val, but everytime I run curl -XGET "localhost:15050/_node/stats/events?pretty" I get different IN count. Please check screenshot below. So how do I determine if total events are fetched?

enter image description here

Upvotes: 0

Views: 1258

Answers (1)

Val
Val

Reputation: 217514

In ES 6.2, the rollover API requires that the index name ends with a sequence number, such as -00001. However, if this is not the case, then it is also possible to use a different index name but you need to specify it explicitly in the rollover call, like this:

POST /agent/_rollover/agent_index_2020_05_13
{
  "conditions": {
    "max_age": "1d"
  }
}

So if the index pointed by the alias agent is older than one day, a new index called agent_index_2020_05_13 will be created by the rollover call.

Note: However, since ES 6.2.4 is EOL, you should upgrade your stack to at least 6.6, then you get ILM support in Logstash for free and it will take care of setting everything up properly in ES.

Upvotes: 1

Related Questions