CODEWITHSUNDEEP
pythonasynchronousconcurrencyconcurrent.futures
leafonsword
leafonsword

Reputation: 3155

What's the difference between concurrent.futures' async and requests-futures' async in python?

I used requests-futures to grab web page asynchronously.And my machine have multicores, so I also want to grab many websites simultaneously,then I try to use concurrent.futures, it seems concurrent.futures also provide asynchronous method,so what's the difference between concurrent.futures' async and requests-futures' async?If they are same,means I could deprecate requests-futures?

Upvotes: 1

Views: 2154

Answers (1)

dano
dano

Reputation: 94931

requests-futures is just a very small wrapper on top of concurrent.futures. You can see this by looking at the source code (With docstrings removed for brevity):

from concurrent.futures import ThreadPoolExecutor
from requests import Session
from requests.adapters import DEFAULT_POOLSIZE, HTTPAdapter

class FuturesSession(Session):

    def __init__(self, executor=None, max_workers=2, *args, **kwargs):
        super(FuturesSession, self).__init__(*args, **kwargs)
        if executor is None:
            executor = ThreadPoolExecutor(max_workers=max_workers)
            # set connection pool size equal to max_workers if needed
            if max_workers > DEFAULT_POOLSIZE:
                adapter_kwargs = dict(pool_connections=max_workers,
                                      pool_maxsize=max_workers)
                self.mount('https://', HTTPAdapter(**adapter_kwargs))
                self.mount('http://', HTTPAdapter(**adapter_kwargs))

        self.executor = executor

    def request(self, *args, **kwargs):
        func = sup = super(FuturesSession, self).request

        background_callback = kwargs.pop('background_callback', None)
        if background_callback:
            def wrap(*args_, **kwargs_):
                resp = sup(*args_, **kwargs_)
                background_callback(self, resp)
                return resp

            func = wrap

        return self.executor.submit(func, *args, **kwargs) # This returns a concurrent.futures.Future

When you use requests-futures, you're really using concurrent.futures.ThreadPoolExecutor, which returns a concurrent.futures.Future when you submit a task to it. If it's more convenient for you to use the API provided by requests-futures for dealing with HTTP requests, it's ok to stick with it, and even use the objects returned by it with other methods provided by the concurrent.futures module.

Upvotes: 5

Related Questions