CODEWITHSUNDEEP
pythonmultithreadingconcurrencyproxyeventlet
Dejell
Dejell

Reputation: 14317

Eventlet threads do not run in parallel

I wrote the following code:

import eventlet
import requests
import redis

redis = redis.StrictRedis(host="localhost", port="6379", db=0)

proxy_1_pool = eventlet.GreenPool(40)

def fetch_items():
    for _ in range(0, 400):
        proxy_1_pool.spawn(fetch_listing)

    proxy_1_pool.waitall()

def fetch_listing():
    logger.info("START fetch: " + str(datetime.utcnow()))
    url_info = redis.spop("listings_to_crawl")
    content = make_request(url_info)
    logger.info("END fetch: " + str(datetime.utcnow()))
    if content:
        do_something(content)

def make_request(url_info):
    r = requests.get(url_info)
    return r.content

def main():
    fetch_items()

Unfortunately I see that fetch_listing is being involved sequentially.

It would always print:

START
END
START 
END

While I would expect to see:

START
START
END 
END

Upvotes: 1

Views: 2226

Answers (1)

temoto
temoto

Reputation: 5577

What's going on:

  • you asked eventlet to execute multiple fetch_listing() concurrently. Parallel as in question title is not going to happen ever, forget about it. And it did as ordered, you can verify by putting eventlet.sleep() right after logger.info...START
  • then execution was blocked by redis.spop and requests.get.

What you do to make blocking code cooperate with eventlet: patching or offload to threadpool.

-import eventlet
+import eventlet ; eventlet.monkey_patch()

Very related questions, highly recommend to read:

Upvotes: 1

Related Questions