CODEWITHSUNDEEP
pythonseleniumselenium-webdriverwebdriver
jake wong
jake wong

Reputation: 5228

Python Selenium clicking next button until the end

This is a follow up question from my first question, and I am trying to scrape a website and have Selenium click on next (until it cant be clicked) and collect the results as well.

This is the html tag from the website with the button:

<div class="pager results-display">
  <span class="action-btn prev inactive" data-page="1">Prev</span>
  <span class="action-btn inactive" data-page="1">1</span>  # Currently in page 1 thus inactive
  <span class="action-btn" data-page="2">2</span>
  <span class="action-btn" data-page="3">3</span> 
  <span class="action-btn" data-page="4">4</span>
  <span class="action-btn" data-page="5">5</span>
  <span class="action-btn" data-page="6">6</span>
  <span class="action-btn" data-page="7">7</span>
  <span class="action-btn" data-page="8">8</span> 
  <span class="action-btn" data-page="9">9</span>
  <span class="action-btn" data-page="10">10</span>
  <span class="action-btn" data-page="11">11</span> 
  <span class="action-btn" data-page="12">12</span>
  <span class="action-btn" data-page="13">13</span>
  <span class="action-btn" data-page="14">14</span>
  <span class="action-btn next" data-page="2">Next</span>
</div>
<div class="no-results-display hidden" style="display: none;">
                    No companies matched your search. Try again.
                </div>

I've tried this code:

elm = driver.find_element_by_name('pager results-display')
elm.click()

And I've also checked this question out, but still wasn't able to solve it.

Any thoughts?

Upvotes: 6

Views: 10619

Answers (1)

Guy
Guy

Reputation: 50864

pager results-display are actually two classes that belongs to the button's parent element. They aren't name attribute, and find_element_by_name gets only one name anyway. Try

elm = driver.find_element_by_class_name('next')
elm.click()

Notice that you might have to relocate the button each time if the DOM has changed after the click.

To click in loop as long as the button is active you can check if the buuton has class inactive

while True:
    elm = driver.find_element_by_class_name('next')
    if 'inactive' in elm.get_attribute('class'):
        break;
    elm.click()

Upvotes: 11

Related Questions