CODEWITHSUNDEEP
c#elasticsearchnest
Paulo Henrique PH
Paulo Henrique PH

Reputation: 361

Scrolling and Paging in Elastic/Nest 6+

Intro

I'm upgrading the elastic version to 6.3 (previously we were using 5.4. Our app is written in C#, thus we use NEST.NET dll to talk with the Elastic server, so we are also updating it to the version 6.0.0.0.

The use case - Before

Until version 5, I was able to execute this query:

jsonStr =" 
   {
      "from": 16224,
      "size": 12,
      "query": {
        "bool": {
          "filter": [
            {
              "bool": {
                "must": [
                  {
                    "terms": {
                      "COMPANY": [
                        "AMP Services Ltd"
                      ]
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    }
}"

Using this NEST/C# code:

Func<SearchRequestParameters, SearchRequestParameters> requestParameters = null;
requestParameters = a => a.Scroll(new TimeSpan(0, 1, 0));
response = Connection.Client.GetInstance().LowLevel.Search<dynamic>("myindex", new PostData<dynamic>(jsonStr), requestParameters);

And with that, I was able to fetch the data without problems,

The use case - NOW

Now, with version 6, I'm trying to execute this very same query:

jsonStr ="
{
  "from": 16224,
  "size": 12,
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [
              {
                "terms": {
                  "COMPANY": [
                    "AMP Services Ltd"
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  }
}"

Using this NEST/C# code (as the previus method signatures are no longer available):

SearchRequestParameters searchRequest = new SearchRequestParameters();
searchRequest.Scroll = new TimeSpan(0, 1, 0);
response = Connection.Client.GetInstance().LowLevel.Search<StringResponse>("myindex", PostData.String(jsonStr), searchRequest);

And I'm getting this error: "Validation Failed: 1: using [from] is not allowed in a scroll context;"

Documentation

I could not find anything in here (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html) and here (https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/sliced-scroll-search-usage.html) to help me replace this logic. Nothing in the forums either.

Do you guys have any insights on this?

Thanks

Upvotes: 1

Views: 3404

Answers (1)

Russ Cam
Russ Cam

Reputation: 125488

It looks to be related to a validation change in Elasticsearch in 6.0.0; In 5.x, the from parameter was allowed for a scroll request but silently ignored. Now in 6.0.0, Elasticsearch is stricter and validates whether from is present for a scroll request and if it is, returns a bad response with an appropriate validation error.

Since a from parameter doesn't make sense for a scroll request, the solution to this is to do one of these two

  1. Remove the from parameter when using the Scroll API
  2. Continue to use the from parameter but do not use the Scroll API.

As an aside, If you are needing to scroll many documents, you may want to use ScrollAll() observable helper to do so.

Upvotes: 3

Related Questions