CODEWITHSUNDEEP
python-3.xseleniumwebdriverwaitwebautomation
Mr. 0xCa7
Mr. 0xCa7

Reputation: 140

How to automate dynamically loaded webpage with Selenium in Python

I've been trying to automate some tasks on a dynamically loaded page using Selenium, and it cannot locate dynamically generated elements by xpath, their value, tags or anything else, it's just like requesting page only fetches source code, without parsing scripts and generating additional content. As a web browser extension, it works really fine, but when exporting to Python module it doesn't. Is there any way to make it work in scripts as intended, just like it works in browser extension?

Say, we have a webpage like this:

<html>
(...)
<body>
  <app-root></app-root>
<script src="REDACTED.js" defer></script><script src="REDACTED.js" nomodule defer></script><script src="REDACTED.js" defer></script><script src="REDACTED.js" defer></script><script src="REDACTED.js" defer></script></body>
</html>

And the browser parses these scripts fine, shows buttons, UI etc. Again, interacting with it using Selenium IDE browser extension does its job. When I am exporting it to Python script it looks like:

# Generated by Selenium IDE
import pytest
import time
import json
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

class TestTest1():
  def setup_method(self, method):
    self.driver = webdriver.Chrome()
    self.vars = {}
  
  def teardown_method(self, method):
    self.driver.quit()
  
  def test_test1(self):
    self.driver.get("https://REDACTED")

    # Logging in to the portal I am interacting with - works fine
    logform = self.driver.find_element_by_id("loginForm")
    logform.find_element_by_name("username").send_keys('REDACTED')
    logform.find_element_by_name("password").send_keys('REDACTED')
    logform.find_element_by_name("login").click()
    
    # Below code does not do anything, as it's not able to find these elements 
    # on the webpage and interact with them
    self.driver.find_element(By.CSS_SELECTOR, ".btn--cta").click()
    self.driver.find_element(By.CSS_SELECTOR, ".create-flow-btn--major > img").click()
    element = self.driver.find_element(By.CSS_SELECTOR, ".create-flow-btn--major > img")
    actions = ActionChains(self.driver)
    actions.move_to_element(element).perform()
    element = self.driver.find_element(By.CSS_SELECTOR, "body")
    actions = ActionChains(self.driver)
    actions.move_to_element(element, 0, 0).perform()
    self.driver.close()
  
p1 = TestTest1()
p1.setup_method(1)
p1.test_test1()
p1.teardown_method(1)

And after running it, the script stops trying to find requested element, and throwing the following exception:

(...) line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".btn--cta"}

Oh Mr. Wu, what shall I do?

Upvotes: 1

Views: 881

Answers (1)

KunduK
KunduK

Reputation: 33384

As I mentioned in my comments you need to use explicit wait to wait for element to be visible on the page.

Use WebDriverWait() and wait for visibility_of_element_located() and your locator.

WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "something")))

You need to import below libraries.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Upvotes: 1

Related Questions