CODEWITHSUNDEEP
pythonseleniumweb-scrapingbeautifulsoup
Void S
Void S

Reputation: 802

Python- WebScraping a page

My code is supposed to go into a website, navigate through 2 pages, and print out all the titles and URL/href within each row.

Currently - My code goes into these 2 pages fine, however it only prints out the first title of each page and not each title of each row.

The page does have some JavaScript, and I think maybe this is why it does not show any links/urls/hrefs within each of these rows? Ideally id like to print the URLS of each row.

from selenium import webdriver
import time

driver = webdriver.Chrome()

for x in range (1,3):
    driver.get(f'https://www.abstractsonline.com/pp8/#!/9325/presentations/endometrial/{x}')
    time.sleep(3)
    page_source = driver.page_source
    eachrow=driver.find_elements_by_xpath("//li[@class='result clearfix']")
    for item in eachrow:
        title=driver.find_element_by_xpath("//span[@class='bodyTitle']").text
        print(title)

Upvotes: 0

Views: 74

Answers (1)

user15398259
user15398259

Reputation:

You're using driver inside your for loop meaning you're searching the whole page - so you will always get the same element.

You want to search from each item instead.

for item in eachrow:
    title = item.find_element_by_xpath(".//span[@class='bodyTitle']").text

Also, there are no "URLs" in the rows as mentioned - when you click on a row the data-id attribute is used in the request.

<h1 class="name" data-id="1989" data-key="">

Which sends a request to https://www.abstractsonline.com/oe3/Program/9325/Presentation/694

Upvotes: 1

Related Questions