CODEWITHSUNDEEP
pythonseleniumfirefoxserverpyvirtualdisplay
Sandzhaj
Sandzhaj

Reputation: 53

Use selenium on webserver with firefox profile

I'm trying to write a script which will sync my data from one site to another. The first site doesn't have public api. But i know the queries, that could provide me json response with all data I need. I decided to use a selenium. The main problem is that i have to be authorized to get this data, but it's too hard to authorize with selenium, because site uses recaptcha2. I also want to use it on my server. So i also use pyvirtualdisplay

I made in my firefox a new profile, then i got authorized on the first site with that profile and used it in the script. Something like this

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

profile = webdriver.FirefoxProfile('/home/admin/.cache/mozilla/firefox/o0eaxyux.user')
browser = webdriver.Firefox(profile, executable_path=r'./geckodriver')
browser.get("https://example.com/p/api/v5/profile/blabla")
response = json.loads(browser.find_element_by_tag_name('body').text)
print(response)

browser.quit()
display.stop()

And it works perfectly on my pc. On the server, pyvirtualdisplay also works if i don't use the profile. But if I use the profile on the server, I get the error:

Traceback (most recent call last):
  File "1.py", line 7, in <module>
    browser = webdriver.Firefox(profile, executable_path=r'./geckodriver')
  File "/home/admin/.local/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
    keep_alive=True)
  File "/home/admin/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/admin/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/admin/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/admin/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: connection refused

geckodriver.log has only one string:

1575543823086   mozrunner::runner   INFO    Running command: "/usr/bin/firefox" "-marionette" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofilegsEa0V"

Any ideas?

Upvotes: 1

Views: 737

Answers (1)

Sandzhaj
Sandzhaj

Reputation: 53

Problem is that i copied firefox profile from another pc. And I have no idea how to fix it. But i found another solution: use cookies instead of profile:

  • Create file
import pickle
from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

profile = webdriver.FirefoxProfile('/home/admin/.cache/mozilla/firefox/o0eaxyux.user')
browser = webdriver.Firefox(profile, executable_path=r'./geckodriver')
browser.get("https://example.com/p/api/v5/profile/blabla")
pickle.dump(browser.get_cookies() , open("cookies.pkl","wb"))

browser.quit()
display.stop()
  • Usage
import pickle
from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

browser = webdriver.Firefox(profile, executable_path=r'./geckodriver')
browser.get("https://example.com/p/api/v5/profile/blabla")
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
    browser.add_cookie(cookie)
browser.get("https://example.com/p/api/v5/profile/blabla")
response = json.loads(browser.find_element_by_tag_name('body').text)
print(response)

browser.quit()
display.stop()

From here: How to save and load cookies using Python + Selenium WebDriver

Upvotes: 1

Related Questions