Python Selenium webscraping returns no data using XPATH

Question

Tried to scrape data from a webpage. After login to the site, in the developer tools able to search the xpath and find the match. But, paython code is not returning the data.

from datetime import datetime
from sqlite3 import Time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import requests
from bs4 import BeautifulSoup
import zerodhacred

browser = webdriver.Chrome("C:/Users/SPILLAIP/Downloads/chromedriver_win32/chromedriver.exe")
browser.get(loginURL)

nifty_bank_values_xpath = browser.find_elements(By.XPATH, "//span[contains(@class, 'pane-legend-item-value__main')]")
    print("Len of nifty_bank_values_xpath: ",len(nifty_bank_values_xpath))

The output is

d:\My Personal\2022\Suresh\Learning\Python\zerodha.py:27: DeprecationWarning: executable_path has been deprecated, please pass in a Service object browser = webdriver.Chrome("C:/Users/SPILLAIP/Downloads/chromedriver_win32/chromedriver.exe")

DevTools listening on ws://127.0.0.1:57153/devtools/browser/74a90941-a12f-4be4-b12a-01b256292a5f [15120:6400:0309/123030.129:ERROR:device_event_log_impl.cc(214)] [12:30:30.129] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F) [15120:6400:0309/123030.137:ERROR:device_event_log_impl.cc(214)] [12:30:30.136] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F) Len of nifty_bank_values_xpath: 0

similarly, when tried with find_element

nifty_bank_values_xpath = browser.find_element(By.XPATH, "//span[contains(@class, 'pane-legend-item-value__main')]")

Getting following error:

raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//span[contains(@class, 'pane-legend-item-value__main')]"} (Session info: chrome=99.0.4844.51) Stacktrace: Backtrace:

Able to find the data in the Dev Tools->Elements returning 6 matches.

image

html captured from dev console


   
   
      
         
             
            
                 
                    
               
                  
                     
                        
                           SS
                           
                        
                        ZX8487
                     
                     
                  
               
            
         
      
      
          
         
              
            
                
               
                  
                     
                  
                  
                     
                        
                           Open
                           33278.9
                        
                        
                           High
                           33890.9
                        
                        
                           Low
                           32948.9
                        
                        
                           Close
                           33158.1
                        
                     
                     
                        
                           Volume
                           —
                        
                        
                           Avg. trade price
                           —
                        
                        
                           Total buy quantity
                           —
                        
                        
                           Total sell quantity
                           —

Prophet · Accepted Answer

You are missing a wait here.
You should wait for the elements to be completely loaded before accessing them with find_elements methods.
The best approach here is to use Expected Conditions explicit waits, as following:

from datetime import datetime
from sqlite3 import Time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import requests
from bs4 import BeautifulSoup
import zerodhacred

browser = webdriver.Chrome("C:/Users/SPILLAIP/Downloads/chromedriver_win32/chromedriver.exe")
wait = WebDriverWait(browser, 20)
browser.get(loginURL)
wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(@class, 'pane-legend-item-value__main')]")))
time.sleep(0.3) #short pause added to make sure that all the relevant elements are loaded, not only the first one
nifty_bank_values_xpath = browser.find_elements(By.XPATH, "//span[contains(@class, 'pane-legend-item-value__main')]")
print("Len of nifty_bank_values_xpath: ",len(nifty_bank_values_xpath))

UPD
Since that element is inside an iframe you have to switch to that iframe before accessing elements inside it, as following:

from datetime import datetime
from sqlite3 import Time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import requests
from bs4 import BeautifulSoup
import zerodhacred

browser = webdriver.Chrome("C:/Users/SPILLAIP/Downloads/chromedriver_win32/chromedriver.exe")
wait = WebDriverWait(browser, 20)
browser.get(loginURL)
wait).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='tradingview']")))

wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(@class, 'pane-legend-item-value__main')]")))
time.sleep(0.3) #short pause added to make sure that all the relevant elements are loaded, not only the first one
nifty_bank_values_xpath = browser.find_elements(By.XPATH, "//span[contains(@class, 'pane-legend-item-value__main')]")
print("Len of nifty_bank_values_xpath: ",len(nifty_bank_values_xpath))

When you finished working with elements inside the iframe, to switch to default content you will need to perform

driver.switch_to.default_content()

Python Selenium webscraping returns no data using XPATH

Answers (2)

Related Questions