CODEWITHSUNDEEP
pythonhtmlseleniumselenium-webdriverwebdriver
Rohan Baisantry
Rohan Baisantry

Reputation: 131

Page Source is not reflecting the changes after submitting a form through selenium

I have created a script which will fill the form and submit it. the website then displays the results.

once i open chrome using selenium, i get the driver.page_source and it gives the correct html output of the initial state.

If i use the driver.page_source after submitting the form,i am only getting the source of the initial state again, that is: no change is reflected even though there is a change in the html.

Question: How do i get the HTML output of the page with changes after submitting the form?

Thanks for the help in advance! ps: i'm new so yeah..

EDIT: I found the answer, it was working fine all the while, but the web page hadn't fully loaded yet and hence i was still getting the old source code, so i just made the driver wait before extracting the new source. thank you!

Upvotes: 3

Views: 2685

Answers (2)

undetected Selenium
undetected Selenium

Reputation: 193338

Once you submit the form before you pull out the page_source to check for the change, it is worth to mention that though the WebClient may have achieved 'document.readyState' equal to "complete" at a certain stage and Selenium gets back the control of program execution, that doesn't guarantees that all the associated Javascript and Ajax Calls on the new page have completed. Until and unless the Javascript and Ajax Calls associated with the DOM Tree gets completed the page is not completely rendered you may not be able to track the intended changes.

An ideal way to check for changes would be to induce WebDriverWait in-conjunction with expected_conditions clause set as title_contains as follows :

driver.find_element_by_xpath("xpath_of_element_changes_page").click()
WebDriverWait(browser, 10).until(EC.title_contains(("full_or_partial_text_of_the_new_page_title")))
source = driver.page_source

Note : While Page Title resides within the <head> tag of the HTML DOM a better solution would be to induce WebDriverWait for the visibility of an element which will be present in all situations within the <body> tag of the DOM Tree as follows :

driver.find_element_by_xpath("xpath_of_element_changes_page").click()
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.ID, "id_of_element_present_in_all_situation")))
source = driver.page_source

Upvotes: 1

Alex K.
Alex K.

Reputation: 855

You can pass Selenium's current page to the scrapy Selector and use usual css and/or xpath selectors to get data from it:

sel_response = Selector(text=driver.page_source.encode('utf-8'))
sel_response.css(<your_css_selector>).extract()

Upvotes: 1

Related Questions