CODEWITHSUNDEEP
pythonseleniumweb-scraping
SK Singh
SK Singh

Reputation: 163

Selenium scraping Stops when I switch to a different tab

I have to scrape product details from an Indian e-commerce site. After displaying the first 20 items, we need to scroll down to get the next set of items. I am using selenium for this in python which opens a tab to scrape the site. But when I switch from this tab to another tab or window, then scrolling stops immediately and am unable to scrape further. This part of the code does scrolling for me.

arrow = browser.find_element_by_xpath('//div[@id="see-more-products" and @class="show-more btn"]')
arrow.click()

Is there any way to keep scraping without being active on that tab?

Upvotes: 0

Views: 447

Answers (2)

haydso
haydso

Reputation: 36

You can try to run selenium headlessly, meaning it scrapes data without actually opening a browser window.

Check this documentation/guide: https://duo.com/decipher/driving-headless-chrome-with-python

Upvotes: 0

Mate Mrše
Mate Mrše

Reputation: 8444

You need to tell Selenium to switch tabs before continuing.

Here is an example from here:

from selenium import webdriver
import time
driver = webdriver.Chrome(executable_path="C:\\chromedriver.exe")
driver.get("https://accounts.google.com/signup")
driver.find_element_by_link_text("Help").click()
#prints parent window title
print("Parent window title: " + driver.title)
#get current window handle
p = driver.current_window_handle
#get first child window
chwnd = driver.window_handles
for w in chwnd:
   #switch focus to child window
   if(w!=p):
   driver.switch_to.window(w)
   break
time.sleep(0.9)
print("Child window title: " + driver.title)
driver.quit()

Upvotes: 1

Related Questions