CODEWITHSUNDEEP
pythonmultithreadingalgolia
groeterud
groeterud

Reputation: 127

How to multithread using Algolia browse_objects

I'm having issues figuring out how to deal with multithreading the ObjectIterator returned with the browse_objects.

If I use the search method the below works:

pool=ThreadPool()
res=index.search(qry)
pool.map(someFunc,res['hits])

which allows me to multithread some function on each element of the list. Unfortunately, Algolia search limits the response to 1000 elements, and I can't change that for other production reasons.

Now with browse_objects, it returns an iterator that pool.map doesn't work on, so I have to run the operations in a single thread like this:

res=index.browse_objects(qry)
for hit in res:
    someFunc(hit)

I know that iterators are supposed to work with map, also tried imap and pool.async_task, but neither kinda works.

I might be missing the obvious here, but I'm not too experienced with working with multithreading...

Upvotes: 0

Views: 86

Answers (1)

groeterud
groeterud

Reputation: 127

So you can draw out the hits by just doing 1 step of the iterator:

hits=[]
for hit in res:
    hits=res.raw_response['hits']
    break

Which in turns provides a list you can map over with pool. Thus you can multithread 1k results at a time and fetch them.

Upvotes: 0

Related Questions