CODEWITHSUNDEEP
pythonpython-3.xelasticsearch
kolurbo
kolurbo

Reputation: 538

helpers.scan() doesn't scroll - ElasticSearch

I am unable to scan whole ElasticSearch index.

ElasticSearch version: 1.7.5

I am trying to scan index but it always fail after 1000 documents fetched. 

res = helpers.scan(es, index="twitter", query={"query": {"match_all": {}}},
                   request_timeout=60)

for hit in res:
    do_some_stuff()

Traceback:

WARNING:elasticsearch:GET http://example.com:9200/_search/scroll?scroll=5m [status:400 request:0.002s]
WARNING:elasticsearch:DELETE http://example.com:9200/_search/scroll [status:400 request:0.001s]
Traceback (most recent call last):
  File "/project/.env/lib/python3.5/site-packages/elasticsearch/helpers/__init__.py", line 301, in scan
    resp = client.scroll(scroll_id, scroll=scroll, request_timeout=request_timeout)
  File "/project/.env/lib/python3.5/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/project/.env/lib/python3.5/site-packages/elasticsearch/client/__init__.py", line 1024, in scroll
    params=params, body=body)
  File "/project/.env/lib/python3.5/site-packages/elasticsearch/transport.py", line 312, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/project/.env/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request
    self._raise_error(response.status, raw_data)
  File "/project/.env/lib/python3.5/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: <exception str() failed>

When I tried manually send request:

POST example.com:9200/twitter/_search?scroll=1m

{
    "size": 100,
    "query": {
        "match_all": {}
    }
}

It returned me 100 documents and scroll_id. But this scroll_id was 524 character long. Probably this can be a problem.

Upvotes: 1

Views: 1229

Answers (1)

kolurbo
kolurbo

Reputation: 538

I have installed newestest elastic package client. So I downgraded from 6.1.1 to 1.7.0

ElasticSearch in version 1.7.0 doesn't support JSON in POST request for scroll id but newer versions support it.

Upvotes: 1

Related Questions