CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingscrapy
Aleksandar Ciric
Aleksandar Ciric

Reputation: 124

scraping "View more" button with scrapy

I am trying to scrape the data from this page https://hipages.com.au/find/electricians/wa/perth, but there is a "view more" button. And I don't know which url to use for the more data.

I inspected the page, opened the network tab, but I was unable to find a good way to scrape all the data with Scrapy.

I know I can use Selenium, but there are more then 200 000 urls, so I can not use Selenium for this, because it would take weeks to get all the data.

Upvotes: 1

Views: 241

Answers (1)

Ryan
Ryan

Reputation: 2183

The website uses an API, therefore you could use it. Ex: https://hipages.com.au/api/directory/sites?suburb=perth&state=wa&category=8&page=1&perpage=10&code=111d887415230e233b23fdaae8e160d62715d99d7c417a33c1ca27c0b47b3a6ce810db7af2cec77fe5a629d12ad9cc68

You can then manipulate the query parameters of the url and then fetch the data via Scrapy.

Upvotes: 2

Related Questions