CODEWITHSUNDEEP
perl
user9148262
user9148262

Reputation: 63

Perl LWP::Simple::get($url) does not work for some urls

I am using LWP::Simple::get($url) library to access data from web pages. The problem is that the get function is not working for the below url. Below is the code snippet:

#!/usr/bin/perl

use LWP::Simple;
use JSON;
use Data::Dumper;

my $url = "https://www.cryptopia.co.nz/api/GetCurrencies";

my $json = get( $url);
die "Could not get $url!" unless defined $json;

my $decoded_json = decode_json($json);
print Dumper($decoded_json);

After running this code it gives the below error:

Could not get https://www.cryptopia.co.nz/api/GetCurrencies!

When I replace the url with : $url = "https://api.coinmarketcap.com/v1/ticker/" it works fine. Please can you tell me what is the root cause and how I can fix it. Also the url mentioned in the code snippet worked once and now suddenly it does not work.

Upvotes: 0

Views: 2160

Answers (2)

ahmnouira
ahmnouira

Reputation: 3411

Just installing the LWP::Protocol::https module fixed my issue.

sudo cpan LWP::Protocol::https

Upvotes: 1

Steffen Ullrich
Steffen Ullrich

Reputation: 123320

This is essentially the same question as this one, only for Perl LWP instead of NodeJS.

The problem is that the target site has a broken setup and is missing an intermediate certificate. This can be seen from the SSLabs report:

This server's certificate chain is incomplete. Grade capped to B.

To work around this you have two secure and one insecure option. The insecure but, unfortunately, often proposed option is to disable all certificate validation. Don't do this because it makes your application vulnerable against man-in-the-middle attacks.

The next option is to have your own trust store which includes all necessary certificates, i.e. both the root CA and also the missing intermediate CA:

use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
    SSL_ca_file => 'myca.pem', 
    # disable OCSP stapling since it results in problems with this site
    SSL_ocsp_mode => SSL_OCSP_NO_STAPLE
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;

myca.pem is in this case the concatenation of the PEM representation for the certificates of the intermediate "COMODO RSA Extended Validation Secure Server CA" and the root "COMODO RSA Certification Authority". I've provided it at Pastebin, here.

The third option is to trust this specific leaf certificate only. When used this will trust a server using this certificate, no matter if the certificate is expired, revoked, the hostname does not match the URL or (as in this case) the trust chain cannot be built because of a missing intermediate certificate:

use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;

my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
    SSL_fingerprint => 'sha256$70bca153ac950b8fa92d20f04dceca929852c42dc1d51bdc3c290df256ae05d3',
    SSL_ocsp_mode => SSL_OCSP_NO_STAPLE,
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;

The fingerprint you see here is the one you can also see in the browser when looking at the certificate.

Upvotes: 5

Related Questions