CODEWITHSUNDEEP
pythonseleniumyoutubeselenium-chromedriver
lee
lee

Reputation: 65

youtube page scroll down using selenium, python

url_main = "https://www.youtube.com/feed/history"

I want to scroll down infinitely in the webpage above until all contents are visible, so I tried

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import time

driver = webdriver.Chrome(driverpath)
wait = WebDriverWait(driver, 10)


driver.get(url_main)
login = driver.find_element_by_css_selector('a[href="https://accounts.google.com/ServiceLogin?service=youtube&uilel=3&passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Faction_handle_signin%3Dtrue%26app%3Ddesktop%26hl%3Dko%26next%3Dhttps%253A%252F%252Fwww.youtube.com%252Ffeed%252Fhistory&hl=ko&ec=65620"]')
login.click()
login = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'input[type="email"]')))
login.send_keys(email)
login.send_keys(Keys.RETURN)
login = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'input[type="password"]')))
login.send_keys(password)
login.send_keys(Keys.RETURN)

content = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'div[id="primary"] div[id="contents"]')))

no_of_pagedowns = 1

while no_of_pagedowns:
    content.send_keys(Keys.PAGE_DOWN)
    time.sleep(1)
    no_of_pagedowns-=1

But not working. I also tried copy-pasting the code in this link but it didn't work at all also.

Upvotes: 1

Views: 902

Answers (1)

NationBoneless
NationBoneless

Reputation: 298

You can implement something like this:

# save the max scrollHeight of the page
scrollHeight = d.execute_script("return window.scrollMaxY")
scrolled_pages = 0
# while we have not reached the max scrollHeight
while d.execute_script("return window.pageYOffset") < scrollHeight:
# scroll one page down
    d.execute_script("window.scrollByPages(1)")
    scrolled_pages += 1
# wait to load any lazy loaded images (may not be needed, depending on your usecase)
    time.sleep(0.2)

Upvotes: 1

Related Questions