Elasticsearch Curator

Question

I am using ElasticSearch Version 1.7.2. I want to take a snapshot with help of the curator. I followed a link get to know that it gives me an incremental snapshot but I didn't get any incremental snapshot of my index.

I have 1 index test having 50,000 docs. I have taken a snapshot of that index. After inserting 100,000 docs in the same index, I again took a snapshot of that index. But when I check it, it's showing me the same data in both my snapshots.

What's the difference between those? I don't think it is incremental in any way. Please help me out.

PS: I only want to backup the remaining 100,000, not the whole data.

Output of GET /_snapshot/my_backup/_all:

{
 "snapshots": [
   {
     "snapshot": "curator-20160509052605",
     "version_id": 1070199,
     "version": "1.7.1",
     "indices": [
        "jal"
     ],
     "state": "SUCCESS",
     "start_time": "2016-05-09T05:26:05.735Z",
     "start_time_in_millis": 1462771565735,
     "end_time": "2016-05-09T05:26:06.282Z",
     "end_time_in_millis": 1462771566282,
     "duration_in_millis": 547,
     "failures": [],
     "shards": {
        "total": 5,
        "failed": 0,
        "successful": 5
     }
  },
  {
     "snapshot": "curator-20160509055355",
     "version_id": 1070199,
     "version": "1.7.1",
     "indices": [
        "jal"
     ],
     "state": "SUCCESS",
     "start_time": "2016-05-09T05:53:55.824Z",
     "start_time_in_millis": 1462773235824,
     "end_time": "2016-05-09T05:53:56.737Z",
     "end_time_in_millis": 1462773236737,
     "duration_in_millis": 913,
     "failures": [],
     "shards": {
        "total": 5,
        "failed": 0,
        "successful": 5
     }
  },
  {
     "snapshot": "curator-20160509060002",
     "version_id": 1070199,
     "version": "1.7.1",
     "indices": [
        "jal"
     ],
     "state": "SUCCESS",
     "start_time": "2016-05-09T06:00:02.282Z",
     "start_time_in_millis": 1462773602282,
     "end_time": "2016-05-09T06:00:03.602Z",
     "end_time_in_millis": 1462773603602,
     "duration_in_millis": 1320,
     "failures": [],
     "shards": {
        "total": 5,
        "failed": 0,
        "successful": 5
     }
   }
 ]
 }

Answer 1

{
 "snapshots": [
   {
     "snapshot": "curator-20160509052605",
     "version_id": 1070199,
     "version": "1.7.1",
     "indices": [
        "jal"
     ],
     "state": "SUCCESS",
     "start_time": "2016-05-09T05:26:05.735Z",
     "start_time_in_millis": 1462771565735,
     "end_time": "2016-05-09T05:26:06.282Z",
     "end_time_in_millis": 1462771566282,
     "duration_in_millis": 547,
     "failures": [],
     "shards": {
        "total": 5,
        "failed": 0,
        "successful": 5
     }
  },
  {
     "snapshot": "curator-20160509055355",
     "version_id": 1070199,
     "version": "1.7.1",
     "indices": [
        "jal"
     ],
     "state": "SUCCESS",
     "start_time": "2016-05-09T05:53:55.824Z",
     "start_time_in_millis": 1462773235824,
     "end_time": "2016-05-09T05:53:56.737Z",
     "end_time_in_millis": 1462773236737,
     "duration_in_millis": 913,
     "failures": [],
     "shards": {
        "total": 5,
        "failed": 0,
        "successful": 5
     }
  },
  {
     "snapshot": "curator-20160509060002",
     "version_id": 1070199,
     "version": "1.7.1",
     "indices": [
        "jal"
     ],
     "state": "SUCCESS",
     "start_time": "2016-05-09T06:00:02.282Z",
     "start_time_in_millis": 1462773602282,
     "end_time": "2016-05-09T06:00:03.602Z",
     "end_time_in_millis": 1462773603602,
     "duration_in_millis": 1320,
     "failures": [],
     "shards": {
        "total": 5,
        "failed": 0,
        "successful": 5
     }
   }
 ]
 }

Answer 2

It's not about Curator that takes incremental snapshots, it's about Elasticsearch. And what "incremental" means is not a difference in documents (the data itself). Incremental refers to segments (files) on disk.

Also, Elasticsearch periodically merges segments. Meaning it's taking a series of smaller (or larger) segments of almost the same size and merges them in a bigger segment. After the bigger segment is done being created, ES is deleting the smaller segments.

So, after you added 100000 documents, it's very likely that ES already merged your old segments (the ones containing the first 50k documents) and now the set of segments might be completely new. In this case, of course that the new snapshot almost doesn't contain any old segments.

To really test this, after you are done with your indexing, take a snapshot. Then add 10 documents or so, or change 10 of them, and then take another snapshot. The new snapshot should have a very small size.