ELK 7.17.1 does not see snapshots in the repo, which were restored back from tape

Question

My ELK7 has a repository (NFS) to which the snapshots are moved after 7 days. Then the snapshots are moved to tape after 90 days. And when i need an old snapshot i make a request to restore the archive to NFS and then restore the archive to ELK (Kibana). It worked like that for some time.

If my snapshot was not yet moved to tape it can be restored to ELK with no problems.

One day a new person started restoring archives from tape to NFS and now I cannot restore my archives which were restored from tape to ELK, it simply does not find the archive. Though if i use DevTools to query i can find the archives.

So the operation is like that: I requested restoration from tape. Files restored.

I check the NFS, files exist:

elasticsearch@pod:/mnt/backup$ ll | grep hHP21J6uTUqLBDc9ERYaiA
-rwxrwxrwx      1 elasticsearch elasticsearch      256 Apr 10  2024 meta-hHP21J6uTUqLBDc9ERYaiA.dat*
-rwxrwxrwx      1 elasticsearch elasticsearch      424 Apr 10  2024 snap-hHP21J6uTUqLBDc9ERYaiA.dat*

I check the repo:

GET /_snapshot

i get:

{
  "backup" : {
    "type" : "fs",
    "settings" : {
      "location" : "/mnt/backup/"
    }
  }
}

if i check all indexes:

GET /_snapshot/backup/_all?ignore_unavailable=true&verbose=false

i can find my snapshot among all the others from DevTools:

{
  "snapshot" : "my-app-2024-04-11",
  "uuid" : "Xgy_ms9nSl2-1oQdOxlUFw",
  "repository" : "backup",
  "indices" : [
    "my-app-2024-04-10"
  ],
  "data_streams" : [ ],
  "state" : "SUCCESS"
},

Now i run check if ELK can see the very snapshot:

GET /_snapshot/backup/my-app-2024-04-11

and get:

{
  "snapshots" : [
    {
      "snapshot" : "my-app-2024-04-11",
      "uuid" : "Xgy_ms9nSl2-1oQdOxlUFw",
      "repository" : "backup",
      "version_id" : 7170199,
      "version" : "7.17.1",
      "indices" : [
        "my-app-2024-04-10"
      ],
      "data_streams" : [ ],
      "include_global_state" : false,
      "state" : "SUCCESS",
      "start_time" : "2024-04-11T22:03:37.442Z",
      "start_time_in_millis" : 1712873017442,
      "end_time" : "2024-04-11T22:10:35.796Z",
      "end_time_in_millis" : 1712873435796,
      "duration_in_millis" : 418354,
      "failures" : [ ],
      "shards" : {
        "total" : 1,
        "failed" : 0,
        "successful" : 1
      },
      "feature_states" : [ ]
    }
  ],
  "total" : 1,
  "remaining" : 0
}

So i try to restore my archive:

POST /_snapshot/backup/my-app-2024-04-10/_restore
{
    "indices": "*,-.*",
    "ignore_unavailable": true,
    "include_global_state": false,
    "rename_pattern": "(.+)",
    "rename_replacement": "archive-$1"
  }

and get:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "snapshot_missing_exception",
        "reason" : "[backup:my-app-2024-04-10/hHP21J6uTUqLBDc9ERYaiA] is missing"
      }
    ],
    "type" : "snapshot_missing_exception",
    "reason" : "[backup:my-app-2024-04-10/hHP21J6uTUqLBDc9ERYaiA] is missing",
    "caused_by" : {
      "type" : "no_such_file_exception",
      "reason" : "/mnt/backup/indices/CkfPwAiNRmanVPJ9LOAHdw/meta-hHP21J6uTUqLBDc9ERYaiA.dat"
    }
  },
  "status" : 404
}

As i read, ELK is looking for this "/mnt/backup/indices/CkfPwAiNRmanVPJ9LOAHdw/meta-hHP21J6uTUqLBDc9ERYaiA.dat" when it cannot find the snap-* and meta-* files in root repo directory.

I came to an idea to reindex my repository, and GPT suggested me this command:

POST /_snapshot/backup/_reload

but i get this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "invalid_snapshot_name_exception",
        "reason" : "[backup:_reload] Invalid snapshot name [_reload], must not start with '_'"
      }
    ],
    "type" : "invalid_snapshot_name_exception",
    "reason" : "[backup:_reload] Invalid snapshot name [_reload], must not start with '_'"
  },
  "status" : 400
}

So my problem is that ELK can see the index but cannot restore it. I could not find a wa y to reindex my repo. Please help to find the cause!!!