CODEWITHSUNDEEP
perllwp
GraveRaven
GraveRaven

Reputation: 49

No content from LWP request

SOLVED: I fixed it by installing HTML::HeadParser. But I still have no idea why it just suddenly stopped working.

I noticed some LWP request that have worked before had stopped working so I made a small script to check why. For some reason I don't get anything back as content from most sites (The only two that have worked so far is icanhazip.com and the gdata.youtube.com api). I get a status 200 back and I get the right Content-Lenght for the header. I have tried fetching the page in both curl and wget with no problems. And if I check the requests with tcpdump I do indeed get a reply with all the right information. I have tried with both LWP::UserAgent and LWP::Simple with the same results.

#!/usr/bin/perl
use warnings;
use strict;
use LWP::UserAgent;

my $agent = LWP::UserAgent->new;
$agent->agent("Mozilla/5.0");

my $req = HTTP::Request->new(GET => 'http://stackoverflow.com');
my $res = $agent->request($req);

print $res->status_line, "\n";
print $res->header("Content-Length"), "\n";
print $res->content, "\n";

Dumping $res with DumpLex gives the following:

$HTTP_Response1 = bless( {
                    _content => '',
                    _headers => bless( {
                                  "cache-control"
                                               => 'public, max-age=15',
                                  "client-aborted"
                                               => 'die',
                                  "client-date"
                                               => 'Sat, 23 Jun 2012 15:07:23 GMT',
                                  "client-peer"
                                               => '64.34.119.12:80',
                                  "client-response-num"
                                               => 1,
                                  connection   => 'close',
                                  "content-length"
                                               => 211684,
                                  "content-type"
                                               => 'text/html; charset=utf-8',
                                  date         => 'Sat, 23 Jun 2012 15:07:22 ',
                                  expires      => 'Sat, 23 Jun 2012 15:07:38 ',
                                  "last-modified"
                                               => 'Sat, 23 Jun 2012 15:06:38 GMT ',
                                  vary         => '*',
                                  "x-died"     => 'Can\'t locate HTML/HeadParser.pm in @INC (@INC contains: /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl .) at /usr/share/perl5/site_perl/LWP/UserAgent.pm line 663.'
                                }, 'HTTP::Headers' ),
                    _msg     => 'OK',
                    _protocol
                             => 'HTTP/1.1',
                    _rc      => 200,
                    _request => bless( {
                                  _content => '',
                                  _headers => bless( {
                                                "content-type" => 'application/html',
                                                "user-agent"   => 'Mozilla/5.0'
                                              }, 'HTTP::Headers' ),
                                  _method  => 'GET',
                                  _uri     => \do { my $v = 'http://stackoverflow.com/' },
                                  _uri_canonical
                                           => 'V: $HTTP_Response1->{_request}{_uri}'
                                }, 'HTTP::Request' ),
                    default_add_content
                             => 1
                  }, 'HTTP::Response' );
$HTTP_Response1->{_request}{_uri_canonical} = $HTTP_Response1->{_request}{_uri};
bless( $HTTP_Response1->{_request}{_uri}, 'URI::http' );

Upvotes: 4

Views: 1691

Answers (2)

Borodin
Borodin

Reputation: 126762

In circumstances like this, when a module seems to be misbehaving beyond its documented functionality, it would be useful to force a reinstallation with

cpan -f -i LWP::UserAgent

and I suggest you do that right now in case there are others problems with your module library.

Upvotes: 3

Ωmega
Ωmega

Reputation: 43703

Script you have in your question is working just fine. I bet you have some issues with library.

I suggest you to run

perl -MCPAN -e 'install LWP::UserAgent'

to ensure your library is up-to-date.

Upvotes: 1

Related Questions