CODEWITHSUNDEEP
pythonseleniumxpathwebdriverwebdriverwait
Jacek Błocki
Jacek Błocki

Reputation: 563

Selenium does not render html page source

I've tried to get data form a web page using python and selenium:

from selenium import webdriver
driver=webdriver.Firefox()
driver.get('https://covid19.min-saude.pt/ponto-de-situacao-atual-em-portugal/')
'RECUPERADOS' in driver.page_source
False

String 'RECUPERADOS' is present in browser window opened by driver, but not in page_source. What prevents selenium form rendering page html? Is there any way to get data from page like the one above?

Upvotes: 1

Views: 1508

Answers (1)

KunduK
KunduK

Reputation: 33384

The element text you are searching for is inside an iframe. You need to switch to iframe first and then wait for element to be visible and then capture the page_source.

Use WebDriverWait() and wait for frame_to_be_available_and_switch_to_it()

Use WebDriverWait() and wait for visibility_of_element_located()

driver.get('https://covid19.min-saude.pt/ponto-de-situacao-atual-em-portugal/')
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//*[@id='main']//iframe[1]")))
WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.XPATH,"//*[name()='text'][text()='RECUPERADOS']")))

if 'RECUPERADOS' in driver.page_source:
       print("YES")
else:
       print("NO")

Import below libraries.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

To jump out from iframe you need to use below code.

driver.switch_to.default_content()

Upvotes: 1

Related Questions