CODEWITHSUNDEEP
pythonmultithreadingperformanceweb-scrapingbeautifulsoup
jshen
jshen

Reputation: 72

Speed up the number of page I can scrape via threading

I'm currently using beautifulsoup to scrape sourceforge.net for various project information. I'm using the solution in this thread. It works well, but I wish to do it yet faster. Right now I'm creating a list of 15 URLs, and feed them into the run_parallel_in_threads. All the URLs are sourceforge.net links. I'm currently getting about 2.5 pages per second. And it seems that increasing or decreasing the number of URLs in my list doesn't have much effect on the speed. Are there any strategy to increase the number of page I can scrape? Any other solutions that are more suitable for this kind of project?

Upvotes: 0

Views: 439

Answers (1)

Devarsh Desai
Devarsh Desai

Reputation: 6112

You could have your threads which run in parallel simply retrieve the web content. Once the html page is retrieved, pass the page into a queue which have multiple workers each parsing a single html page. Now you've essentially pipelined your workflow. Instead of having each thread do multiple steps (retrieve page, scrape, store). Each of your threads in parallel simple retrieve the page and then have it pass the task into a queue which processes these tasks in a round robin approach.

Please let me know if you have any questions!

Upvotes: 2

Related Questions