CODEWITHSUNDEEP
seleniumwebdriverphantomjs
JBJ
JBJ

Reputation: 1109

Selenium returns different html source than viewed in browser

Im trying to use Selenium to load next page with results by clicking Load More button from this site. However the source code of the html page loaded by selenium does not show(load) actual products which one can see when browsing. Here is my code:

from selenium import webdriver      
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import os
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

#browser = webdriver.Firefox()#Chrome('./chromedriver.exe')
URL = "https://thekrazycouponlady.com/coupons-for/costco"
PATIENCE_TIME = 60
LOAD_MORE_BUTTON_XPATH = '//button[@class = "kcl-btn ng-scope"]/span' 
caps = DesiredCapabilities.PHANTOMJS
# driver = webdriver.Chrome(r'C:\Python3\selenium\webdriver\chromedriver_win32\chromedriver.exe')
caps["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
driver = webdriver.PhantomJS(r'C:\Python3\selenium\webdriver\phantomjs-2.1.1-windows\bin\phantomjs.exe',service_log_path=os.path.devnull,desired_capabilities=caps)
driver.get(URL)

while True:
    try:
        time.sleep(20)
        html = driver.page_source.encode('utf-8')
        print(html)
        loadMoreButton = driver.find_element_by_xpath(LOAD_MORE_BUTTON_XPATH)


        loadMoreButton.click()

    except Exception as e:
        print (e)
        break
print ("Complete")

driver.quit()

Not sure if I can attach sample html file here for reference. Anyway, what is the problem and how do I load exactly the same page with selenium as i do via browser?

Upvotes: 0

Views: 1404

Answers (1)

Guy
Guy

Reputation: 50809

It might be due to the use of PhantomJS, it isn't maintained any more and deprecated from Selenium 3.8.1. Use Chrome headless instead.

options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)

Upvotes: 2

Related Questions