CODEWITHSUNDEEP
pythonseleniumweb-scrapingbeautifulsoupmechanize
ashishkumar148
ashishkumar148

Reputation: 1005

How to approach web-scraping in python

I am new to python just started on python web-scraping. I have to scrape data from this realtor site

I need to scrape all the details op read-state agents according to their real-state agency; For this on the web-browser I have to follow the following instructions

  1. Go to this site
  2. click on agency offices button, enter 4000 pin in search box and then submit.
  3. then we get list of the agencies.
  4. go to our team tab and then we get agents their.
  5. then we have to go to each agents page and record their information.

Can anyone tell me how to approach this. Whats the best way to make this type of scrapers.

Do i have to use selenium for the interaction with the pages.

I have worked on request, BeautifulSoup and simple form submit using mechanize

Upvotes: 0

Views: 234

Answers (2)

Asim Mittal
Asim Mittal

Reputation: 101

You might wanna give Node and Jquery a try. I used to use Python all the time, but it gets messy and hard to maintain after a while.

Using node, you can turn the page HTML into a DOM object and then scrape all the data very easily using Jquery. I have done this for imdb here: “Using JQuery & NodeJS to scrape the web” @asimmittal https://medium.com/@asimmittal/using-jquery-nodejs-to-scrape-the-web-9bb5d439413b

You can modify this to scrape yelp

Upvotes: 1

AceLewis
AceLewis

Reputation: 353

I would recommend on a searching site that you either use Selenium or Requests with sessions, the advantage of Selenium it it will probably work however it will be slow. For Selenium you should just use the Selenium IDE (Firefox add on) to record what you do then get the HTML from the webpage and use beautifulsoup to parse the data.

If you want to scrape the data quickly and without using much resources I usually use Requests with Sessions. To scrape a website like this you should open up a modern web browser (Firefox, Chrome) and use the network tools for that browser (usually located in developer tools or via right click inspect element). Once you are recording the network you can interact with the webpage to see the connections made to the server. In an example search they may use suggestions e.g

https://suggest.example.com.au/smart-suggest?query=4000&n=7&regions=false

The response then will probably be a JSON of the suggested results. Once you select a suggestion you can just submit a request with that search parameters e.g

https://www.example.com.au/find-agent/agents/petrie-terrace-qld-4000

The URLs for the agents will the be in that HTML page, you just then need to separately send a request to each page to get the information using BeautifulSoup.

Upvotes: 1

Related Questions