CODEWITHSUNDEEP
pythonpython-3.x
luke
luke

Reputation: 61

Python: running multiple web requests in parallel

I'm new to Python and I have a basic question but I'm struggling to find an answer online, because a lot of the examples online seem to refer to deprecated APIs, so sorry if this has been asked before.

I'm looking for a way to execute multiple (similar) web requests in parallel, and retrieve the result in a list.

The synchronous version I have right now is something like:

urls = ['http://example1.org', 'http://example2.org', '...']

def getResult(urls):
  result = []
  for url in urls:
    result.append(get(url).json())
  return result

I'm looking for the asynchronous equivalent (where all the requests are made in parallel, but I then wait for all of them to be finished before returning the global result).

From what I saw I have to use async/await and aiohttp but the examples seemed way too complicated for the simple task I'm looking for.

Thanks

Upvotes: 3

Views: 3899

Answers (1)

Mohamed Ahmed Nabil
Mohamed Ahmed Nabil

Reputation: 4069

I am going to try to explain the simplest possible way to achieve what you want. Im sure there are more cleaner/better ways to do this but here it goes.

You could preform what you want using the python "threading" library. You can use it to create separate threads for each request and then run all the threads concurrently and get an answer.

Since you are new to python, to simplify things further I am using a global list called RESULTS to store in the results of the get(url) rather than returning them from the function.

import threading

RESULTS=[] #List to store the results

#Request Single Url Result and store in global RESULTS
def getSingleResult(url):
    global RESULTS
    RESULTS.append( ( url, get(url).json()) )

#Your Original Function
def getResult(urls)
    ths=[]
    for url in urls:
        th=threading.Thread(target=getSingleResult, args=(url,)) #Create a Thread
        th.start() #Start it
        ths.append(th) #Add it to a thread list

    for th in ths:
        th.join() #Wait for all threads to finish

The usage of the global results is to make it easier rather than collecting results from the threads directly. If you wish to do that you can check out this answer How to get the return value from a thread in python?

Of course one thing to note that multi-threading in python doesnt provide true parallelism but rather concurrency especially if you are using the standard python implementation due to what is known as the Global Interpreter Lock

However for your use case it would still provide for you the speed up you need.

Upvotes: 1

Related Questions