CODEWITHSUNDEEP
.netelasticsearchnestbulk
LLS
LLS

Reputation: 2228

Retrieving an unknown amount of documents in Elasticsearch using NEST

I am trying to figure out how to retrieve possibly a large amount of documents using NEST 2.3.2. Because there is a limit on the number of documents in a response (set by the server admin), I want to use Scroll to be safe.

I've read this but I didn't find a way to achieve that using NEST 2.3.2 library. This documentation is for 1.x, and it doesn't work for 2.x.

Specifically, when I try to set SearchType = Elasticsearch.Net.SearchType.Scan in the SearchRequest, it automatically sends request to /_search?search_type=scan and the server just returns "search_phase_execution_exception" because "Scroll must be provided when scanning...". But I don't know how to get a Scroll ID.

Does anyone know how to achieve this? I want to scroll over the qualified documents with possibly several requests until they are all retrieved. If anyone can provide some sample code that would be very helpful. Thank you very much.

Upvotes: 0

Views: 136

Answers (1)

ASN
ASN

Reputation: 1883

I tried something like this for my problem in which I need to get all the documents indexed and then do some processing. I used scroll. You can have a look at my question from here

var scanResults = client.Search<IndexName>(s => s
                .From(0)
                .Size(20) //any size you can give
                .MatchAll()
                .Source(so => so
                .Include(fi => fi.Field(fieldName))
                .SearchType(Elasticsearch.Net.SearchType.Scan)
                .Scroll("5m") //time for which the scrollId is saved. Can be anything.
            );

            var results = client.Scroll<IndexName>("10m", scanResults.ScrollId);

            while (results.Documents.Any())
            {
                //do whatever you want to do from the resultant documents

                results = client.Scroll<IndexName>("10m", results.ScrollId);
            }

More information about scroll

Hope this helps.

Upvotes: 2

Related Questions