Selenium with firefox self signed certificate issue, cannot bypass

Question

Let me preface by saying I am definitely novice and apologies if I improperly describe or name something. I am trying to automate the interactions with a web page served up by an application on a server in my local subnet, not over the internet, and it is using a self signed cert.

I am on windows 10, using the pycharm IDE with Python 3.9.10, Selenium 3.141.0, Firefox 95.0.2 and the latest geckodriver version 0.30.0.

I have browsed the many posts with fixes for this issue. I started with Selenium 4.1 but learned that everything the fixes used, like desired capabilities, was deprecated and replaced with service and options objects that I just couldn't get to work.

So I downgraded to Selenium 3.141.0 to try and implement the fixes suggested.

First, when starting with a simple webdriver instantiation without any options other than pointing to the driver, I got a popup for login. I could not inspect it so I figured it was not an html element.

Login prompt

With some googling i found it to be basic browser based authentication and bypassed it by adding <user>:<pass>@<address> URL format. This then got me past the login prompt and actually to the landing page of the webUI. It was still an unsecure connection and you can see in the following screenshot, it said there was a security exception was added.

Security exception

However, within a couple of seconds of the page loading I got a popup that there was no repsonse from the server, which I knew was not the case, as I could manually access the site and accept the cert security warning to access the site.

No response from server

If I reloaded the page manually, or with a refresh from selenium, I then got the certificate error page.

Certificate warning

I have tried any combination of the below fixes I have found in various posts, using options, profile and capabilities parameters, and all of them give me the same result as I explained above when I use nothing but a webdriver instance.

options = Options()
options.add_argument("--ignore-certificate-errors")
options.accept_insecure_certs = True

profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True
profile.assume_untrusted_cert_issuer = True

capabilities = DesiredCapabilities().FIREFOX.copy()
capabilities["acceptSslCerts"] = True
capabilities["acceptInsecureCerts"] = True

browser = webdriver.Firefox(executable_path="<path>", firefox_profile=profile, capabilities=capabilities, options=options)

Any help is greatly appreciated, and if anyone knows a fix for Selenium 4.1 using service and options objects let me know, I am willing to try that too. But looking at the documentation and code for the firefox options class, it looks like there is no longer an add_arguments function, Options doc, as I think they have moved solely to set_preferences. As for the service object that every error told me to use due to deprecation, looking at that Service doc the only function that looked promising was the command_line_args, but looking at the code, it simply raises a NotImplementedError..... That is why I gave up on getting anything to work with Selenium 4.1, because I could not find any preferences that would work, as that seemed like the only thing that could be modified in the options object.

Answer 1

Looking into the geckodriver logs I found that the --ignore-certificate-errors arg was being passed to the firefox call. I only found these logs because they were in the IDE project files, or else I would have never thought to look at them.

In the log file, you can see that there is a warning that a settings file doesn't exist. I am using the FirefoxProfile class without sending a path to an existing profile, wherein Selenium will create its own temporary profile. I think this profile is causing issues, as passing the path to the default firefox profile solved the issue.

geckodriver log