CODEWITHSUNDEEP
pythonweb-crawler
Evg
Evg

Reputation: 3080

python asyncore or threadpool for web crawler?

It seem what i can do fast crawler with python in two ways:

  1. thread pool with block sockets

  2. non block sockets select,asyncore,etc..

i thnk where is no real need in thread here, and solution #2 better.

which is better and why?

Upvotes: 2

Views: 1003

Answers (1)

Alex B
Alex B

Reputation: 84822

Twisted is usually preferred to asyncore. It is an asynchronous I/O framework that can also work with thread pools.

In Python, you should prefer asynchronous IO to threads, simply because threads are a second class citizen in its canonical implementation (CPython) due to GIL.

Upvotes: 3

Related Questions