CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scrapingselenium-firefoxdriver
SSC Fan
SSC Fan

Reputation: 135

Selenium with Firefox: FirefoxProfile stops working when used a second time

I am trying to use selenium with firefox to create a profile of a browser I want to use repeatedly. I start by loading it up with extensions I want it to have, then call a function with that profile as a parameter to use for scraping.

profile = webdriver.FirefoxProfile()
profile.add_extension(extension='extension/1/path')
profile.add_extension(extension='extension/2/path')


def sele_scrape(profile,url):

  options = Options()
  options.headless = True

  driver = webdriver.Firefox(firefox_profile=profile,options=options,executable_path='path/to/geckodriver')

  driver.get(url)
  source=driver.page_source
  driver.quit()


sele_scrape(profile,'url1')
sele_scrape(profile,'url2')

It doesn't matter what order the urls are in, or what urls I use. The second one always causes this error:

File "(the python file for this project)", line 37, in <module>
    sele_scrape(profile,'url2')
  File "(the python file for this project)"
    driver = webdriver.Firefox(firefox_profile=profile,options=options,executable_path='path/to/geckodriver')

File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/webdriver.py", line 166, in __init__
    capabilities.update(options.to_capabilities())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/options.py", line 180, in to_capabilities
    opts["profile"] = self._profile.encoded
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 173, in encoded
    self.update_preferences()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 102, in update_preferences
    self._write_user_prefs(self.default_preferences)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 226, in _write_user_prefs
    with open(self.userPrefs, "w") as f:

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/8j/lrckmwqs5bs9b4srrjbj6r5c0000gn/T/tmpiluv90lz/user.js'

This line

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/8j/lrckmwqs5bs9b4srrjbj6r5c0000gn/T/tmpiluv90lz/user.js'

Is particularly bizarre to me because I have no what its trying to refer to or why.

Any help, or even just a direction that I can look in, would be much appreciated!

Upvotes: 0

Views: 1478

Answers (1)

furas
furas

Reputation: 142804

Probably when you quit() then it removes profile - and you would have to create again profile.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url):

    profile = webdriver.FirefoxProfile()
    #profile.add_extension(extension='extension/1/path')
    #profile.add_extension(extension='extension/2/path')
    
    options = Options()
    #options.headless = True
    
    driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')
    
    driver.get(url)
    source = driver.page_source
    driver.quit()

# --- main ---

sele_scrape('https://stackoverflow.com')
sele_scrape('https://stackoverflow.com')

Code works correctly if I use close() instead of quit()

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url, profile, options):
   
    driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')
    
    driver.get(url)
    source = driver.page_source
    driver.close()

# --- main ---

profile = webdriver.FirefoxProfile()
#profile.add_extension(extension='extension/1/path')
#profile.add_extension(extension='extension/2/path')

options = Options()
#options.headless = True

sele_scrape('https://stackoverflow.com', profile, options)
sele_scrape('https://stackoverflow.com', profile, options)

Frankly, if profile should be reused then I would create driver only once and I would skip close().

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url, driver):
   
    driver.get(url)
    source = driver.page_source

# --- main ---

profile = webdriver.FirefoxProfile()
#profile.add_extension(extension='extension/1/path')
#profile.add_extension(extension='extension/2/path')

options = Options()
#options.headless = True

driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')

sele_scrape('https://stackoverflow.com', driver)
sele_scrape('https://stackoverflow.com', driver)

driver.close()

Upvotes: 1

Related Questions