CODEWITHSUNDEEP
scrapy
Evan Zamir
Evan Zamir

Reputation: 8481

Having trouble getting xpath to select an Next button

I'm trying to crawl this gem website:

https://www.irocks.com/search?_token=q57It5iOxH0R1TpCusPK781faIVHprh47BexHVkM&code=&collection=&description=&interval=&locality=&max=&min=&mode=advanced&name=&operator=%3E%3D&query=&species=&status%5B0%5D=available&status%5B1%5D=on-hold

There's been some weird stuff going on and I can't figure out how to get certain elements like the href in the Next button.

For example,

response.xpath('//section') yields:

[<Selector xpath='//section' data='<section class="specimen-details">\n\t<...'>,
 <Selector xpath='//section' data='<section class="specimen-related hidd...'>,
 <Selector xpath='//section' data='<section class="shows hidden-print">\n...'>,
 <Selector xpath='//section' data='<section class="blog hidden-print">\n ...'>,
 <Selector xpath='//section' data='<section class="navigation">\n        ...'>]

But when I look in the console I see an additional <section class="specimen-list"> that does not show up there and contains the navigation buttons within it. I'm not sure what's going on. Any help or advice appreciated!

Upvotes: 1

Views: 22

Answers (1)

Sagun Shrestha
Sagun Shrestha

Reputation: 1198

The xpath to get href of next page is //a[@rel="next"]/@href

So you can basically do

response.xpath('//a[@rel="next"]/@href').get()

or using css selector

response.css('a[rel="next"]::attr(href)').get()

get() method works in newer version of scrapy, if it doesn't works in your use extract_first().

Upvotes: 1

Related Questions