CODEWITHSUNDEEP
pythonseleniumweb-scraping
Khaled Koubaa
Khaled Koubaa

Reputation: 527

Get rid of a pop up window while infinite scrolling using selenium

I want to perform infinite scroll down on this page: https://www.financialjuice.com/home

after some scrolls down, a window appear (sign up), and I need to remove it so the scroll down comlete, but I can't, I created this code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

import time

u = "https://www.financialjuice.com/home"
driver = webdriver.Chrome(executable_path=r"C:\chromedriver.exe")
driver.get(url)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")    
driver.implicitly_wait(60) # seconds
    
print('start scrolling')
for i in range(5):
    driver.find_element_by_css_selector('html').send_keys(Keys.END)
    print("scroll", i)
    time.sleep(5)
# the pop up window will appear after the 4th scroll

# the below code to try to click on the page to remove it
el=driver.find_elements_by_xpath('//*[@id="aspnetForm"]/div[3]/header/nav/div[2]/div/div[1]/div/a[1]/img[2]')[0]

action = webdriver.common.action_chains.ActionChains(driver)
# action.move_to_element_with_offset(el, 5, 5)
action.move_to_element_with_offset(el, 5, 0)

action.click()
action.perform()

### after remove it, I complete the scrolls:
for i in range(100):
    driver.find_element_by_css_selector('html').send_keys(Keys.END)
    print("scroll", i)
    time.sleep(1)

I need solution to this pop up so I can scroll down infinitely on this web page

Upvotes: 0

Views: 392

Answers (1)

Nic Laforge
Nic Laforge

Reputation: 1876

Implement a method to detect if the popover is present and discard it.

def ignore_sign_up_popover():
    sign_up = driver.find_elements_by_css_selector('#signup')
    if len(sign_up) > 0 and sign_up[0].is_displayed():
        TouchActions(driver).tap_and_hold(0, 0).release(0, 0).perform()

Then add the check on each iteration (prior scrolling)

for i in range(5):
    ignore_sign_up_popover()
    driver.find_element_by_css_selector('html').send_keys(Keys.END)
    print("scroll", i)
    time.sleep(5)

Note you will need to import touchActions:

from selenium.webdriver.common.touch_actions import TouchActions

Also there is an issue with w3c/chrome which will raise an exception with the touchAction, if I remember correctly there is a bug opened on Chrome side for it. Some info here

As mentioned, you can disable it for the time being with:

options = webdriver.ChromeOptions()
options.add_experimental_option('w3c', False)
driver = webdriver.Chrome(options=options)

Upvotes: 2

Related Questions