How to enable scroll functionality in elastic search

Question

I have a web url api which serving by the elastic search.

• My web urls is https://data.emp.com/employees
• I have 50 employee(data) in my index
• On each scroll 7 employee will be adding take 7,14,21..49,50
• On each scroll 7 employee will appear first then 14 employees, ..49,50 employees
• My Api below URL is all 50 employees at one shot

    def elastic_search():
        """
         Return full search using match_all
        """
        try:
     
            full_search= es.search(index="employees",scroll = '2m',size = 10,body={ "query": {"match_all": {}}})
            hits_search = full_search['hits']['hits']
            return hits_search 
        except Exception as e:
            logger.exception("Error" + str(e))
            raise

I have modified above code like below

        sid =  search["_scroll_id"]
        scroll_size = search['hits']['total']
        scroll_size = scroll_size['value']
        # Start scrolling
        while (scroll_size > 0):

            #print("Scrolling...")
            page = es.scroll(scroll_id = sid, scroll = '1m')

            #print("Hits : ",len(page["hits"]["hits"]))
            
            # Update the scroll ID
            sid = page['_scroll_id']
        
            # Get the number of results that we returned in the last scroll
            scroll_size = len(page['hits']['hits'])
            search_text = page['hits']['hits']
            print (search_text)

My api is throwing [] because my last search_text giving blank. In the log it is printing each set of 7 employees. But My web url api is loading loading and last it showing blank page

Please help in updating in returning "hits_search" which is in elastic_search function

Answer 1

I guess elasticsearch from and size will do the trick for you if you have doc less than ≤ 10k. But still if you want to use the scroll API then this is what you need,

    # declare a filter query dict object
    match_all = {
        "size": 7,
        "query": {
            "match_all": {}
        }
    }

    # make a search() request to get all docs in the index
    resp = client.search(
        index = 'employees',
        body = match_all,
        scroll = '2s' # length of time to keep search context
    )
    
    # process the first 7 documents here from resp
    # iterate over the document hits for each 'scroll'
    for doc in resp['hits']['hits']:
        print ("\n", doc['_id'], doc['_source'])
        doc_count += 1
        print ("DOC COUNT:", doc_count)
    
    # keep track of pass scroll _id
    old_scroll_id = resp['_scroll_id']

    # use a 'while' iterator to loop over document 'hits'
    while len(resp['hits']['hits']):

        # make a request using the Scroll API
        resp = client.scroll(
            scroll_id = old_scroll_id,
            size = 7,
            scroll = '2s' # length of time to keep search context
        )

        # iterate over the document hits for each 'scroll'
        for doc in resp['hits']['hits']:
            print ("\n", doc['_id'], doc['_source'])
            doc_count += 1
            print ("DOC COUNT:", doc_count)

See Ref: https://kb.objectrocket.com/elasticsearch/how-to-use-python-to-make-scroll-queries-to-get-all-documents-in-an-elasticsearch-index-752