Elasticsearch index nodes data inconsistent in a way we can't explain

Question

We are experiencing issues with inconsistent data between nodes, giving rise to significantly indeterministic search results. We are on ES version 8.15.0.

So far we think we've ruled out the following:

• Sort order: we use a deterministic sort order whose last fallback is document id
• Split-brain problem (https://trifork.nl/blog/how-to-avoid-the-split-brain-problem-in-elasticsearch/)
  • We have three nodes, all of which are master eligible (one of them is voting only)
  • If I read this correctly this might not be an issue any more? https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch

Our first hypothesis is that this happens due to deleted documents: https://www.elastic.co/guide/en/elasticsearch/reference/current/consistent-scoring.html. This is supported by the fact that the nodes appear to have the same number of documents but different number of deleted ones.

Using dfs_query_then_fetch does not, however, solve the issue, which hints that it’s not that simple. Also, looking into the results from /_explain, we see that “number of documents containing term” differs between the nodes for the same terms.

We could solve the indeterminacy using ?prefer=... in the search, but seeing as the search result from one of the nodes is noticeably better than the other this seems less than ideal. Also we’d like to understand where the difference came from in the first place.

Any idea what to look into?