CODEWITHSUNDEEP
pythoncssseleniumxpathweb-scraping
Owais Arshad
Owais Arshad

Reputation: 313

Selenium - identifying web element

I am using Python to scrape data from a website. While I have been able to use Selenium to log in, I cannot identify the search field once logged in. It appears the web page loads with frames (not iframes), but I cannot access the frame with the search field.

I have tried changing the frame to the relevant frame (which seems to work - no error is thrown up) but then if I try searching for the search element by CSS / Xpath / Name / id I get a NoSuchElementException. I am using the Chrome webdriver.

Any suggestions? The page source is as follows:

  <html>
  <head>
    <title> XYZ </title>
    <meta http-equiv="content-type" content="text/html; charset=utf-8" />
    <meta http-equiv="Content-Script-Type" content="text/javascript" />
    <meta http-equiv="Content-Style-Type" content="text/css" />
    <meta http-equiv="content-language" content="en" />
    <script type="text/javascript">
    if (navigator && navigator.appVersion && navigator.appVersion.match("Safari") && !navigator.appVersion.match("Chrome")) {
        // hack to force a window redraw
        window.onload = function() {
        document.getElementsByTagName('html')[0].style.backgroundColor = '#000000';
        }
    }
    </script>
  </head>

  <frameset id="wc-frameset" rows="82,*" frameborder="no" border="0" framespacing="0">
    <frame frameborder="0" src="/frontend/header/" name="top" marginwidth="0" marginheight="0" scrolling="no" noresize="noresize" />
      <frameset cols="*,156,850,*" frameborder="NO" border="0" framespacing="0">
        <frame frameborder="0" src="/frontend/fillbar/" name="fillbar" marginwidth="0" marginheight="0" scrolling="no"  noresize="noresize" />
        <frame frameborder="0" src="/frontend/navigation/" name="navigation" marginwidth="0" marginheight="0" scrolling="no"  noresize="noresize" />
        <frame frameborder="0" src="/frontend/frames/" name="content_area" marginwidth="0" marginheight="0" scrolling="no" noresize>
        <frame frameborder="0" src="/frontend/fillbar/" name="fillbar" marginwidth="0" marginheight="0" scrolling="no"  noresize="noresize" />
      </frameset>
  </frameset>
</html>

The code that I have so far is:

username = driver.find_element_by_id("username")
password = driver.find_element_by_id("password")
username.send_keys("****")
password.send_keys("****")   
driver.find_element_by_class_name("bg-left").click()
#this bit works

driver.switch_to_frame("content_area")
#this seems to work too, got the frame name from the page source

search = driver.find_element_by_id("field-name")
search.send_keys("TEST")
#this fails, no element found

The target frame source code is:

      <div id="field-name" class="field field-StringField">
  <label for="name">Name</label>            <div class="input-con"><input id="name" name="name" type="text" value=""></div>
      </div>

Upvotes: 2

Views: 266

Answers (2)

StrikerVillain
StrikerVillain

Reputation: 3776

It is possible that there are duplicate elements in the page. Try the following in chrome:

  1. Open url in chrome
  2. Open developer tools F12
  3. Press ESC to open the chrome console
  4. Select your frame enter image description here
  5. Search for similar elements using xpath in console $x("//input[@id='name']")

This should list the number of elements.

Upvotes: 1

Jayesh Doolani
Jayesh Doolani

Reputation: 1233

Maybe you need to wait for the page to load up completely before continuing searching the element. You can try something like:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

driver.switch_to_frame("content_area")

try:
    # this line adds wait for the element to be visible 
    WebDriverWait(driver, 10).until(EC.visibility_of_element_located(By.ID, 'name'))
except TimeoutException:
    # display page timed out error

search = driver.find_element_by_id("name")
search.send_keys("TEST")

Upvotes: 1

Related Questions