CODEWITHSUNDEEP
sortingelasticsearchscrollnest
GMD
GMD

Reputation: 761

elasticseach 2.4 : retrieve all records which are fulfilling all search criterias using scroll

I am using elastic search for the first time and based on requirements i have some doubts and questions for scroll

To retrieve all data which are fulfilling all search criteria 1)I am trying to use scroll but i found while searching about it https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_21_search_changes.html i found Search type scan is deprecated but NEST is supporting it so should i use "search type scan" or "sort by doc"? (I am using elastic search 2.4)

2)Can i use "sorting on any field" when using scrolling?

3)while doing clear scroll var test2 = client.ClearScroll(x=>x.ScrollId(results.ScrollId));

Getting error as below: Invalid NEST response built from a unsuccessful low level call on DELETE: /_search/scroll

Audit trail of this API call:

{"scroll_id":["c2NhbjswOzE7dG90YWxfaGl0czoxMjs="]}

Response:

{}

so is it correct way of clearing scroll or not?

Update: : below is my code :

 List<Object> indexedList = new List<Object>();
    ISearchResponse<ListingSearch> listingResult =
            client.Search<ListingSearch>(search => search
                  .Index(Constant.ES_INDEX)                          
                  .Type(Constant.ES_TYPE)
                  .From(listingSearch.StartIndex)
                  .Size(10)
                  .Source(s => s.Include(i => i.Fields(outpputFields)))                         
                  .Query(query => query.
                            Bool(boolean => boolean.
                            Must(
                                must => must.Term(t => t.Field("is_deleted").Value(false))

                )
                 .Sort(x => x.Field("_doc", SortOrder.Ascending))                       
                .Scroll("60s")
                );
List<Object> indexedList = new List<Object>();
var results = client.Scroll<ListingSearch>("60s", listingResult.ScrollId);
                while (results.Documents.Any())
                {
                    foreach (var doc in results.Hits)
                    {
                        indexedList.Add(doc);
                    }

                    results = client.Scroll<ListingSearch>("60s", results.ScrollId);
                }


var test2 = client.ClearScroll(x=>x.ScrollId(results.ScrollId));

//Clear Scroll

With above code I am getting data but if i change size from 10 to 1000, getting no records. Not sure if issue is the amount of data because my ES db has only 12-15 documents.

Upvotes: 2

Views: 473

Answers (1)

Russ Cam
Russ Cam

Reputation: 125518

NEST 2.x versions have SearchType.Scan because NEST 2.x versions are compatible with all Elasticsearch 2.x versions, so the search type needs to exist when using NEST 2.x against Elasticsearch 2.0. Sending the search type through in later versions won't have any effect.

The most efficient way of retrieving documents with scroll is sorting by _doc but you can specify any sort parameters when scrolling.

When using the scroll API, you should use the scroll_id from the previous request in the next scroll call to fetch the next set of results. Once you have finished with a scroll, it is a good idea to clear it by calling ClearScroll() as you are doing. Your call looks correct; perhaps the scroll_id has already expired at the point you make the clear call?

Upvotes: 1

Related Questions