CODEWITHSUNDEEP
pythonsslweb-scrapingcertificate
Eric Brenner
Eric Brenner

Reputation: 51

python - getting SSL error when trying to scrape a webpage

I'm trying to scrape this webpage using Python: https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php

I've been using the requests package. I can "solve" the issue by setting verify=False, however I've read that that's not secure. In other threads, people said to point the requests.get() function to the filepath of the relevant certificate. I exported the certificate from my browser, and then tried that, but with no luck. This

requests.get('https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php',verify='C:/Users/ericb/Desktop/fftoolboxscoutfantasysportscom.crt')

gives the SSL error still

SSLError: HTTPSConnectionPool(host='fftoolbox.scoutfantasysports.com', port=443): Max retries exceeded with url: /football/rankings/PrintVersion.php (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))

And this

requests.get('https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php',cert='C:/Users/ericb/Desktop/fftoolboxscoutfantasysportscom.crt')

yields

Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]

I've done a decent amount of webscraping before, but I've never had to deal with certificates until now. How can I get around this? I should also note that I'd like to put my final Python script and any files it uses onto a public GitHub repo. But I don't want do do anything that would jeopardize my security, like uploading keys or something.

Upvotes: 1

Views: 2148

Answers (1)

Patrick Mevzek
Patrick Mevzek

Reputation: 12495

The server is misconfigured, it does not send the intermediate certificate it needs to send. See this report: https://www.ssllabs.com/ssltest/analyze.html?d=fftoolbox.scoutfantasysports.com&hideResults=on

Certificates provided 1 (1776 bytes)

Chain issues Incomplete

Or https://sslanalyzer.comodoca.com/?url=fftoolbox.scoutfantasysports.com

Trusted by Microsoft? No (unable to get local issuer certificate) UNTRUSTED

Trusted by Mozilla? No (unable to get local issuer certificate) UNTRUSTED

With openssl s_client -connect fftoolbox.scoutfantasysports.com:443 -showcerts you can see:

Certificate chain
 0 s:/OU=Domain Control Validated/CN=fftoolbox.scoutfantasysports.com
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2

And the webserver should be configured to send the /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2 intermediary certificate but it does not.

So, you could contact the website and tells them they are misconfigured. You will not be the only one impacted by that, as the second link shows.

Alternatively, you could add the missing certificate locally as fully trusted, but this kind of lowers your security. You can also download the missing certificate (not the one of the website, the intermediary one) locally and add verify=/path/to/certificate in your requests.get call.

Upvotes: 1

Related Questions