CODEWITHSUNDEEP
phpelasticsearchcrondelete-file
O Connor
O Connor

Reputation: 4392

Delete log data from Elasticsearch that is older than 1 day automatically using crontab

Due to a lot of log data coming into my small server I would like to delete Elasticsearch log data every day using crontab. I have done so many researches on the Internet and have found the solution using curator or Elasticsearch _ttl. But nothing works for me. Now I have no clue any more how to google or what to google. Because I have tried many things what I have found on Google but nothing works. So please help me with a clear instruction.

Upvotes: 2

Views: 4440

Answers (1)

untergeek
untergeek

Reputation: 863

Elasticsearch Curator can only delete indices, not individual documents stored in indices.

If you are trying to delete indices, and your index names meet the expected criteria, then Curator will do the job.

Curator expects indices to have a prefix and a time string. For example --prefix logstash- --timestring %Y.%m.%d would match an index named logstash-2014.07.30

A Curator command to delete indices matching these criteria:

  • Connect to elasticsearch host es-host
  • time unit of days
  • indices older than 1 time-unit
  • a prefix of prod-
  • a time string year.month.day (%Y.%m.%d) might look like this:

curator --host es-host delete indices --older-than 1 --time-unit days --prefix prod- --timestring %Y.%m.%d

TTL is for letting Elasticsearch delete documents after they have lived a given time. This is problematic for logging use-cases for the same reasons that a million SQL statements like delete from TABLE where datestamp < 2014.06.01 is more costly than dropping a partitioned table, e.g. drop table DATA-2014.05. There is a lot of disk I/O involved, and it can hurt performance. If you are not continuously indexing hundreds (or more) documents per second, then TTLs may still be a good fit for you. See more about that here.

Upvotes: 4

Related Questions