CODEWITHSUNDEEP
pythonseleniumweb-scraping
Aytida
Aytida

Reputation: 189

Unable to identify what to 'click' for next page using selenium

I am trying to get search results from yahoo search using python - selenium and bs4. I have been able to get the links successfuly but I am not able to click the button at the bottom to go to the next page. I tried one way, but it could't identify after the second page.

Here is the link:

https://in.search.yahoo.com/search;_ylt=AwrwSY6ratRgKEcA0Bm6HAx.;_ylc=X1MDMjExNDcyMzAwMgRfcgMyBGZyAwRmcjIDc2ItdG9wLXNlYXJjaARncHJpZANidkhMeWFsMlJuLnZFX1ZVRk15LlBBBG5fcnNsdAMwBG5fc3VnZwMxMARvcmlnaW4DaW4uc2VhcmNoLnlhaG9vLmNvbQRwb3MDMARwcXN0cgMEcHFzdHJsAzAEcXN0cmwDMTQEcXVlcnkDc3RhY2slMjBvdmVyZmxvdwR0X3N0bXADMTYyNDUzMzY3OA--?p=stack+overflow&fr=sfp&iscqry=&fr2=sb-top-search

This is what im doing to get data from page but need to put in a loop which changes pages:

page = BeautifulSoup(driver.page_source, 'lxml')
lnks = page.find('div', {'id': 'web'}).find_all('a', href = True)

for i in lnks:
    print(i['href'])

Upvotes: 0

Views: 209

Answers (2)

Tarek Rahman
Tarek Rahman

Reputation: 88

You don't need to scroll down to the bottom. The next button is accessible without scrolling. Suppose you want to navigate 10 pages. The python script can be like this:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
driver=webdriver.Chrome()
driver.get('Yahoo Search URL')
# Let's create a loop containing the XPath for next button
# As well as waiting for the next button to be clickable.
for i in range(10):
    WebDriverWait(driver, 5).until(EC.element_to_be_clickable(By.XPATH, '//a[@class="next"]'))
    navigate = driver.find_element_by_xpath('//a[@class="next"]').click()

Upvotes: 1

Prophet
Prophet

Reputation: 33361

The next page button is on the bottom of the page so you first need to scroll to that element and then click it. Like this:

from selenium.webdriver.common.action_chains import ActionChains

actions = ActionChains(driver)
next_page_btn = driver.find_element_by_css_selector("a.next")
actions.move_to_element(next_page_btn).build().perform()
time.sleep(0.5)
next_page_btn.click()

Upvotes: 1

Related Questions