CODEWITHSUNDEEP
elasticsearchcurllogstashkibanaelasticsearch-curator
ADARSH K
ADARSH K

Reputation: 626

How to delete data from a specific index in elasticsearch after a certain period?

I have an index in elasticsearch with is occupied by some json files with respected to timestamp. I want to delete data from that index.

curl -XDELETE http://localhost:9200/index_name

Above code deletes the whole index. My requirement is to delete certain data after a time period(for example after 1 week). Could I automate the deletion process?

I tried to delete by using curator.

But I think it deletes the indexes created by timestamp, not data with in an index. Can we use curator for delete data within an index?

It will be pleasure if I get to know that either of following would work:

References are taken from the official site of elasticsearch.

Thanks a lot in advance.

Upvotes: 6

Views: 13339

Answers (3)

georgep68
georgep68

Reputation: 86

Simple example using Delete By Query API:

POST index_name/_delete_by_query
{
  "query": {
    "bool": {
      "filter": {
        "range": {
          "timestamp": {
            "lte": "2019-06-01 00:00:00.0",
            "format": "yyyy-MM-dd HH:mm:ss.S"
          }
        }
      }
    }
  }
}

This will delete records which have a field "timestamp" which is the date/time (within the record) at which they occured. One can run the query to get a count for what will be deleted.

GET index_name/_search
{
  "size": 1,
  "query: {
-- as above --

Also it is nice to use offset dates 

         "lte": "now-30d",

which would delete all records older than 30 days.

Upvotes: 4

neun24
neun24

Reputation: 233

You can always delete single documents by using the HTTP request method DELETE.

To know which are the id's you want to delete you need to query your data. Probably by using a range filter/query on your timestamp.

As you are interacting with the REST api you can do this with python or any other language. There is also a Java client if you prefer a more direct api.

Upvotes: 0

Enrichman
Enrichman

Reputation: 11337

You can use the DELETE BY QUERY API: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

Basically it will delete all the documents matching the provided query:

POST twitter/_delete_by_query
{
  "query": { 
    "match": {
      "message": "some message"
    }
  }
}

But the suggested way is to implement indexes for different periods (days for example) and use curator to drop them periodically, based on the age:

...
logs_2019.03.11
logs_2019.03.12
logs_2019.03.13
logs_2019.03.14

Upvotes: 5

Related Questions