CODEWITHSUNDEEP
pythonmultithreadingseleniumfirefoxbrowser
MUHAMMED OKUYUCU
MUHAMMED OKUYUCU

Reputation: 41

Close all browsers with selenium when using multithreads

I m doing web scraping with selenium. And i m using multithreads library. My script opens 3 firefox browsers at the same time and scraping. After finished to scraping, i want to close all browsers, i tried many way but Browser.quit() and browser.close() closing 1 browser, other 2 browser do not close.

def get_links():
   some code here...
def get_driver():
   global driver
   driver = getattr(threadLocal, 'driver', None)
   if driver is None:
       chromeOptions = webdriver.ChromeOptions()
       chromeOptions.add_argument("--headless")
       driver = webdriver.Firefox(executable_path)
       setattr(threadLocal, 'driver', driver)
return driver

def get_title(thisdict):
   import datetime
   driver = get_driver()
   driver.get(thisdict["url"])
   time.sleep(5)
   driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")

if __name__ == '__main__':
   ThreadPool(3).map(get_title, get_links())
   driver.close()  #or driver.quit()

Upvotes: 2

Views: 562

Answers (2)

MUHAMMED OKUYUCU
MUHAMMED OKUYUCU

Reputation: 41

I solved the problem with the code below. After Multithread finished all scraping, I m calling closeBrowsers function. And the function kills all open firefox browsers.

import os

def closeBrowsers():
    os.system("taskkill /im firefox.exe /f")

if __name__ == '__main__':
    ThreadPool(2).map(get_title, get_links())
    closeBrowsers()

Upvotes: 1

Aleksander Ikleiw
Aleksander Ikleiw

Reputation: 2685

you have to use the self.selenium.stop() function. The quit()basically calls driver.dispose method which in turn closes all the browser windows. close() closes the browser window on which the focus is set.

Upvotes: 1

Related Questions