CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scraping
Edward
Edward

Reputation: 3

Python web scraping with Selenium on Dynamic Page - Issue with looping to next element

I am hoping someone can please help me out and put me out of my misery. I have recently started to learn Python and wanted to challenge myself with some web-scraping.

Over the past couple of days I have been trying to web-scrape this website (https://ebn.eu/?p=members). On the website, I am interesting in:

  1. Clicking on each logo image which brings up a pop-up
  2. From the pop-up scrape the link which is behind the text "VIEW FULL PROFILE"
  3. Move to the next logo and do the same for each

I have managed to get Selenium up and running but the issue is that it keeps opening the first logo and copying the same link as opposed to moving to the next one. I have tried in various different ways but came up against a brick wall.

My code so far:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

PATH = "/home/ed/Documents/python/chromedriver" # location of the webdriver - amend as requried

url = "https://ebn.eu/?p=members"

driver = webdriver.Chrome(PATH)
driver.get(url)

member_list = []

# Flow to open page and click on link to extract href

members = driver.find_elements_by_xpath('//*[@class="projectImage"]') # Looking for the required class - the image which on click brings up the info

for member in members:
    print(member.text) # to see that loop went to next iteration
    member.find_element_by_xpath('//*[@class="projectImage"]').click()
    wait = WebDriverWait(driver, 10)
    element = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'VIEW FULL PROFILE')))
    links = driver.find_element_by_partial_link_text("VIEW FULL PROFILE")
    href = links.get_attribute("href")
    member_list.append(href)
    member.find_element_by_xpath("/html/body/div[5]/div[1]/button").click()

    print(member_list)
driver.quit()

P.S: I have tried changing the member.find to:

member.find_element_by_xpath('.//*[@class="projectImage"]').click() But then I get "Unable to find element"

Any help is very much appreciated.

Thanks

Upvotes: 0

Views: 253

Answers (1)

Shashank Mistry
Shashank Mistry

Reputation: 78

If you study the HTML of the page, they have the onclick script which basically triggers the JS and renders the pop-up. You can make use of it. You can find the onclick script in the child element img. So your logic should be like (1)Get the child element (2)go to first child element (which is img always for your case) (3)Get the onclick script text (4)execute the script.

child element

for member in members:
    print(member.text) # to see that loop went to next iteration
    # member.find_element_by_xpath('//*[@class="projectImage"]').click()
    
    #Begin of modification
    child_elems = member.find_elements_by_css_selector("*") #Get the child elems
    onclick_script = child_elems[0].get_attribute('onclick')#Get the img's onclick value
    driver.execute_script(onclick_script)                   #Execute the JS
    time.sleep(5)                                           #Wait for some time
    #end of modification 
    wait = WebDriverWait(driver, 10)
    element = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'VIEW FULL PROFILE')))
    links = driver.find_element_by_partial_link_text("VIEW FULL PROFILE")
    href = links.get_attribute("href")
    member_list.append(href)
    member.find_element_by_xpath("/html/body/div[5]/div[1]/button").click()

    print(member_list)

You need to import time module. I prefer time.sleep over wait.until, it's more easier to use when you are starting with web scraping.

Upvotes: 1

Related Questions