CODEWITHSUNDEEP
python-3.xscrapy
Adam Henry
Adam Henry

Reputation: 63

Python Scrapy - Trying to use pagination that is "#" and redirects me to same page

I'm building a scraper for this page I want to follow to next page after making first page work and after setting auto throttle and download speed to be gentle and I tried using:

next_page = response.xpath('//div[@class="global-pagination"]/a[@class="next"]/@href').get()
if next_page is not None:
    yield response.follow(next_page, self.parse)

Problem is that href in that class is # and basically it opens same page again. How do I make it work?

Upvotes: 0

Views: 135

Answers (1)

Moein Kameli
Moein Kameli

Reputation: 976

If you take a look at your browser developer tool you will see when you go to other pages data loads from loadresult. furthermore by searching in Form Data you'll see there is a field named page have a value of the page you requested which you can request any other page by changing it in your formdata in FormRequest. enter image description here

from scrapy.http import FormRequest

FormRequst(url=url, formdata=formdata={'page': <page number>}, callback=<parse method>)

Upvotes: 1

Related Questions