CODEWITHSUNDEEP
perlwww-mechanize
mepstein
mepstein

Reputation: 143

Perl script using WWW::Mechanize to connect to https site just started failing

I have a Perl script that uses WWW::Mechanize to connect to a site over https, and that script just stopped working the other day. The status and error message I get back are 500 and "Can't connect to jobs.illinois.edu:443". The URL I'm trying to connect to is https://jobs.illinois.edu/. I can connect from my browser (Firefox). My platform is Linux -- up-to-date Arch Linux. I can also connect (using WWW::Mechanize) to other https sites.

I tried using LWP::UserAgent, and the behavior is the same.

I'm using ssl_opts => { SSL_version => 'TLSv1' }; I don't remember why I added that -- it may have been necessary to get it working at some point.

Any ideas on how to fix this, or how I might get more information as to what the problem is? Are there other ssl options I can try?

I have a feeling there was some slight configuration change on the site that led to this problem -- maybe some SSL-protocol version change or something like that. (I don't think I updated anything on my machine inbetween the times it worked and stopped working.)

Thanks.

Here's sample code that fails:

#!/usr/bin/perl

use strict;
use warnings;

use constant AJB_URL => 'https://jobs.illinois.edu/academic-job-board';

use WWW::Mechanize;

my $mech = WWW::Mechanize->new( ssl_opts => { SSL_version => 'TLSv1' } );

$mech->get( AJB_URL );

It returns:

Error GETing https://jobs.illinois.edu/academic-job-board: Can't connect to jobs.illinois.edu:443 at ./test2.pl line 12.

Upvotes: 3

Views: 3120

Answers (1)

Steffen Ullrich
Steffen Ullrich

Reputation: 123320

... that script just stopped working the other day.

Which in most cases is caused by server-side or client-side changes. But I assume that you did not make any changes on the client side.

Calling your code with perl -MIO::Socket::SSL=debug4... gives:

DEBUG: ...SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Looking at the SSLLabs report you see two trust paths, where one requires an extra download. The root-CA "USERTrust RSA Certification Authority" for the first trust path is not installed on my system (Ubuntu 14.04), and I guess it is not installed on yours (no information about your OS is known, so just guessing). This means the second trust chain will be used and the relevant Root-CA "AddTrust External CA Root" is also installed on my system. Unfortunately this trust chain is missing an intermediate certificate ("Extra download"), so the verification fails.

To fix the problem, find the missing root-CA which should match the fingerprint 2b8f1b57330dbba2d07a6c51f70ee90ddab9ad8e and use it:

$ENV{PERL_LWP_SSL_CA_FILE} = '2b8f1b57330dbba2d07a6c51f70ee90ddab9ad8e.pem';

Looking at the certificate you see that it was issued on 22 May 2015, i.e. three days ago. This explains why the problem happened just now.

Upvotes: 7

Related Questions