Reputation: 15136
How can I make, say N
url calls in parallel, and process the responses as they come back?
I want to ready the responses and print them to the screen, maybe after some manipulations. I don't care about the order of the responses.
Upvotes: 0
Views: 714
Reputation: 52000
One basic solution that comes to mind is to use threading
.
Depending of the number of URL you retrieve in parallel, you could have one thread per URL. Or (scale better), have a fixed number of "worker" threads, reading the URL from a shared Queue
.
Upvotes: 0
Reputation: 249153
You can use Twisted Python for this, such as in the example here: https://twistedmatrix.com/documents/13.0.0/web/howto/client.html#auto3
Twisted is an asynchronous programming library for Python which lets you carry out multiple actions "at the same time," and it comes with an HTTP client (and server).
Upvotes: 1