CODEWITHSUNDEEP
performanceperlsslhttpswww-mechanize
MikeU
MikeU

Reputation: 114

WWW::Mechanize with SSL works but response is slow

I'm using WWW::Mechanize to make an HTTPS connection to a piece of hardware running its own SSLv3-based web server.

My code works but the responses to get(), submit() and click() come back slowly, ranging from 6 to 12 seconds.

There's a configuration issue that's causing the server's IP address to differ from the one it uses in its certificate. For that reason, I'm ignoring hostname verification for the time being.

That could be the reason why WWW::Mechanize is slow. However, there isn't any slowness when using a regular web browser to perform the same functions. I'm running on Solaris 10 (sparc), with Perl 5.20.1 and OpenSSL 0.7.9d. Mechanize runs fast when doing normal HTTP connections to a different server.

I also tried all of the agent aliases. No improvement. 

use WWW::Mechanize;
$ENV{HTTPS_VERSION}                = 3;    # Try SSLv3 first
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;    # Ignore verification
my $agent = WWW::Mechanize->new( autocheck => 1 );
$agent->get($https_url);
$agent->credentials( $username, $password );
$agent->get($status_url);
print "$agent->{content}\n";
$agent->form_number(2);
$agent->click();
print "$agent->{content}\n";

UPDATE:

I updated OpenSSL to 1.0.1j and recompiled Net::SSLeay 1.66. I also updated IO::Socket::SSL to 2.002. I still saw the same behavior.

I then recompiled LWP::Protocol::https 6.0.6 and WWW::Mechanize 1.73 for good measure. Same behavior.

I added use IO::Socket::SSL qw(debug3); to my script. I can see a 6 second delay in the debug output.

Fetching initial page...
DEBUG: .../IO/Socket/SSL.pm:2537: new ctx 11476520
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:589: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
# >>> There's a 6 second delay here <<<
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:708: ssl handshake done
DEBUG: .../IO/Socket/SSL.pm:2570: free ctx 11476520 open=11476520
DEBUG: .../IO/Socket/SSL.pm:2582: OK free ctx 11476520
Credentials...
Fetching status page...
DEBUG: .../IO/Socket/SSL.pm:2537: new ctx 11476520
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:589: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
# >>> Another 6 second delay <<<
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:708: ssl handshake done
DEBUG: .../IO/Socket/SSL.pm:2570: free ctx 11476520 open=11476520
DEBUG: .../IO/Socket/SSL.pm:2582: OK free ctx 11476520
DEBUG: .../IO/Socket/SSL.pm:2537: new ctx 11476520
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:589: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:640: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:663: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:673: waiting for fd to become ready: SSL wants a read first
# >>> Another 6 second delay <<<
DEBUG: .../IO/Socket/SSL.pm:693: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:653: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:708: ssl handshake done
DEBUG: .../IO/Socket/SSL.pm:2570: free ctx 11476520 open=11476520
DEBUG: .../IO/Socket/SSL.pm:2582: OK free ctx 11476520
Submitting form...
# Repeat of behavior when fetching the initial page, except ctx is 12700768

SOLUTION:

Here's my working script with the changes described in my conversation with Steffen. Keep-alive was the solution.

use WWW::Mechanize;
use IO::Socket::SSL qw(debug3);                     # Turn on level 3 debug
use LWP::ConnCache;
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;             # Ignore verification
my $agent = WWW::Mechanize->new(autocheck => 1);
$agent->conn_cache(LWP::ConnCache->new);            # Enable keep-alive
$agent->get($https_url);
$agent->credentials($username, $password);
$agent->get($status_url);
$agent->form_number(2);
$agent->click();

Upvotes: 2

Views: 1095

Answers (1)

Steffen Ullrich
Steffen Ullrich

Reputation: 123320

There's a configuration issue that's causing the server's IP address to differ from the one it uses in its certificate. For that reason, I'm ignoring hostname verification for the time being.

Better use the SSL_verifycn_name setting of IO::Socket::SSL to define, which name you expect in the certificate.

That could be the reason why WWW::Mechanize is slow.

Probably not, because you are just disabling checks. Disabling does not make it slower, but enabling would not make it slower too because these checks are fast and don't need any additional network activity.

I'm running on Solaris 10 (sparc), with Perl 5.20.1 and OpenSSL 0.7.9d.

I doubt that you are using 0.7.9d, you probably mean 0.9.7d. It is still a very unusual configuration, that is using a modern Perl with a 10 year old version of OpenSSL. I would suggest that you use a current version instead and maybe your problems go away then.

However, there isn't any slowness when using a regular web browser to perform the same functions.

Current browsers use a modern TLS stack which has more efficient ciphers, session resumption etc. Again, try to use a recent version of OpenSSL instead.

Upvotes: 3

Related Questions