CODEWITHSUNDEEP
pythonpython-2.7selenium
Ryflex
Ryflex

Reputation: 5779

Wait for XHR Request and/or a Console message for Selenium in Python

I'm trying to make selenium capture the page source after it has fully rendered, if I go to the page and capture straight away only a bit of the page has rendered, if I put in a sleep of 30 seconds it fully renders but I want it to be more efficient.

If we use https://twitter.com/i/notifications as an example, you'll see that after 5~ seconds after the page loads there is a toast_poll and a timeline XHR request.

I want to be able to detect one of these requests and wait until one fires, then that is an indicator that the page has loaded fully.

The site that I am using fires console.log("Done") so if I could detect the console commands in PhantomJS & Firefox then this would be an even better choice than waiting for an XHR request, just wait until Done appears in the console and then that is the indicator that the page has loaded fully.

Regarding the Duplicate Flagging of this Post:

This question is in regards to PhantomJS and Firefox, the post Detect javascript console output with python is from over a year ago and the answer given only works on Chrome, I am looking for a PhantomJS and Firefox option, which I already think based on StackOverflow isn't possible so that's why my start of my post is about waiting for an XHR request.

I've already tried the following code, but it doesn't work for me.. I get zero response even though the website is throwing a console.log("Done")

from seleniumrequests import PhantomJS
from seleniumrequests import Firefox
from selenium import webdriver
import os

webdriver.DesiredCapabilities.PHANTOMJS['phantomjs.page.customHeaders.User-Agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/37.0.2062.120 Chrome/37.0.2062.120 Safari/537.36'
webdriver.DesiredCapabilities.PHANTOMJS['loggingPrefs'] = { 'browser':'ALL' }

browser = PhantomJS(executable_path="phantomjs.exe", service_log_path=os.path.devnull)
browser = webdriver.Firefox()
browser.set_window_size(1400, 1000)

url = "https://website.com"
browser.get(url)

for entry in browser.get_log('browser'):
    print entry

I'm unable to test with browser = webdriver.Firefox() commented out because I am not sure how to have two lots of DesiredCapabilities set.

Upvotes: 2

Views: 2882

Answers (1)

Florent B.
Florent B.

Reputation: 42538

You could override the console.log function and wait for the "Done" message with execute_async_script:

from selenium import webdriver

driver = webdriver.Firefox()
driver.set_script_timeout(10)

driver.get("...")

# wait for console.log("Done") to be called
driver.execute_async_script("""
  var callback = arguments[0];
  console.log = function(message) {
    if(message === "Done")
      callback();
  };
  """)

Upvotes: 3

Related Questions