Is there a limit on the number of fields specified in the “include” param in Reindex call

Question

This is the error message I am facing while executing the Reindex call

[2020-03-13T16:08:39,042][WARN ][o.e.i.r.TransportReindexAction] [ExportDatabase] giving up on search because it failed with a non-retryable exception org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1188) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1172) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:385) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_212] at java.lang.Thread.run(Unknown Source) [?:1.8.0_212] Caused by: org.elasticsearch.ElasticsearchException$1: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:658) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-6.3.0.jar:6.3.0] ... 26 more Caused by: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:746) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:69) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:39) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:28) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.elasticsearch.common.xcontent.support.XContentMapValues.filter(XContentMapValues.java:174) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceContext.getFilter(FetchSourceContext.java:234) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:132) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:56) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:393) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:368) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0] ... 9 more

So as per my analysis this is the query which I am executing:

POST _reindex
{
"source":{
"index":"index_source",
"size":1000,
"query":
{
"terms":{
"ID":["7078"],
"boost":1.0
}
},
"_source":
{
"includes":here we have 1216 fields]
}
},
"dest":
{
"index":"index_dest"
}
}

Here if I remove the 1216 fields in the include param or make it less to around 500 then it works

So need help in this, is there any other way we can only reindex for these specific fields or is there something wrong I am doing.

Answer 1

From elastic search documentation Mapping

1000 is maximum number of fields allowed in an index. To increase the limit you can update "index.mapping.total_fields.limit"

Warning

The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources. If you increase this setting, we recommend you also increase the indices.query.bool.max_clause_count setting, which limits the maximum number of boolean clauses in a query.