CODEWITHSUNDEEP
pythondjangoseleniumdjango-views
Sashaank
Sashaank

Reputation: 964

How to scrape data faster with selenium and django

I am working on a web scraping project. In this project, I have written the necessary code to scrape the required information from a website using python and selenium. All of this code resides in multiple methods in a class. This code is saved as scraper.py.

When I execute this code, the program takes sometime between 6 and 10 seconds to extract all the necessary information from the website.

I wanted to create a UI for this project. I used django to create the UI. In the webapp, there is a form which when submitted opens a new browser window and starts the scraping process.

I access the scraper.py file in django views, where depending on the form inputs, the scraping occurs. While this works fine, the execution is very slow and takes almost 2 minutes to finish running.

How do I make the execution of the code faster using django faster? can you point me some tutorial on how to convert the scraper.py code into an api that django can access? will this help in making the code faster?

Thanks in advance

Upvotes: 0

Views: 942

Answers (1)

Sudev Suresh Sreedevi
Sudev Suresh Sreedevi

Reputation: 168

Few tiny tips,

  • How is your scraper.py working in the first place? Does it simply print the site links/details, or store it in a text file, or return them? What exactly happens in it?
  • If you wish to use your scraper.py as an "API" write your scraper.py code within a function that returns the details of your scraped site as a dictionary. Django's views.py can easily handle such dictionaries and send it over to your frontend HTML to replace the parts written in Jinja2.
  • Further speed can be achieved (in case your scraper does larger jobs) by using multi-threading and/or multi-processing. Do explore both :)

Upvotes: 2

Related Questions