CODEWITHSUNDEEP
pythonhtmlscrapycss-selectors
Alexander
Alexander

Reputation: 111

Scrapy: How to get the number of pages?

I am trying to get the number of pages on this site, but nothing works.

response.css('._2uKgC p::text').getall()

html:

<nav class="_2uKgC" aria-label="Page navigation" data-qa-target="pagination">
<p class="">Page 2 of 209</p>
<ul>...</ul></nav>

What am I doing wrong?

Upvotes: 0

Views: 331

Answers (1)

gangabass
gangabass

Reputation: 10666

You need to get sessionId and requestId from HTML code and next request this URL:

f"https://www.healthgrades.com/api3/usearch?where=CA&sessionId={sessionId}&requestId={requestId}&sort.provider=bestmatch&source=init&what=Acupuncture&category=provider&cid&debug=false&debugParams=false&isPsr=false&isFsr=false&isFirstRequest=true&pageNum=1&userLocalTime=23%3A55"

After to you parse JSON response you'll find ["search"]["totalPages"] (and ["search"]["totalCount"]) values.

Upvotes: 1

Related Questions