CODEWITHSUNDEEP
pythonbeautifulsoupmechanizeweb-scraping
Lostsoul
Lostsoul

Reputation: 25999

mechanize opening multiple pages

I am using mechanize and beautifulsoup to create a python script to pull some data from a web page. The scraping works fine but the problem I am having is in moving to multiple pages. Is there a way to move between pages in mechanize in a loop?

Here's what I tried

Browser().follow_link(text_regex="Next")

but it goes to the next page which is fine but if the "Next" button doesn't exist it just dies. I'm not sure how to either make a better loop or really just check if the link exists before running the above follow link command.

Most of the examples and documentation I have found seems to only work on one page.

Upvotes: 2

Views: 1327

Answers (1)

phooji
phooji

Reputation: 10385

How does your code "just die"? If it throws an exception, you can catch it and do something to handle it propery (inside your loop as is):

try:
   Browser.follow_link(text_regex="Next")
except Exception:
   print "No more next button; terminating loop (but not dying mysteriously)"
   break

Upvotes: 1

Related Questions