CODEWITHSUNDEEP
perlhttplwplwp-useragent
Lanbo
Lanbo

Reputation: 15712

Send a plain string request with LWP

To get a response from a certain website, I have to give one exact request string, HTTP/1.1. I tried that one with telnet, it gives me the response I want (a redirect, but I need it).

But when I try to give the same request string to HTTP::Request->parse(), I merely get the message 400 URL must be absolute.

I am not sure if it's the website or LWP giving me that, because as I said, the response worked with telnet.

This is the code:

my $req = "GET / HTTP/1.1\n".
  "Host: www.example-site.de\n".
  "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1\n".
  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\n".
  "Accept-Language: en-us,en;q=0.5\n".
  "Accept-Encoding: gzip, deflate\n".
  "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\n".
  "Keep-Alive: 115\n".
  "Connection: keep-alive\n";

# Gives correct request string
print HTTP::Request->parse($req)->as_string;

my $ua = LWP::UserAgent->new( cookie_jar => {}, agent => '' );
my $response = $ua->request(HTTP::Request->parse($req));

# 400 error
print $response->as_string,"\n";

Anyone can help me here?

Upvotes: 0

Views: 2237

Answers (3)

Lumi
Lumi

Reputation: 15314

It looks to me like parsing the request isn't 100 % round-trip-safe, meaning you cannot feed the response back into a request.

Looks like a bug at first sight, but the module's been out for such a long time… On the other hand, I didn't even know you could use this module to parse a request, so maybe it's not so well tested.

The following test case should point you to the problem, which is that the URL isn't properly assembled for being fed to the $req->request method.

use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request;
use Test::More;

my $host = 'www.example.com';
my $url = '/bla.html';

my $req = <<"EOS";
GET $url HTTP/1.1
Host: $host
EOS

# (1) parse the request
my $reqo = HTTP::Request->parse($req);
isa_ok $reqo, 'HTTP::Request';
diag explain $reqo;
diag $reqo->as_string;

# (2) construct the request
my $reqo2 = HTTP::Request->new( GET => "http://$host$url" );
isa_ok $reqo2, 'HTTP::Request';
diag explain $reqo2;
diag $reqo2->as_string;

is $reqo->uri, $reqo2->uri, 'both URLs are identical';

my $ua = LWP::UserAgent->new( cookie_jar => {}, agent => '' );
for ( $reqo, $reqo2 ) {
    my $response = $ua->request( $_ );
    diag $response->as_string,"\n";
}

done_testing;

Upvotes: 0

Lanbo
Lanbo

Reputation: 15712

Ok, I did it using Sockets. After all, I had the HTTP request and wanted the plain response. Here the code for people who are interested:

use IO::Sockets;

my $sock = IO::Socket::INET->new(
    PeerAddr => 'www.example-site.de',
    PeerPort => 80, 
    Proto => 'Tcp',
);
die "Could not create socket: $!\n" unless $sock;

print $sock, $req;

while(<$sock>) {
    # Look for stuff I need
}

close $sock;

It's just important to remember to leave the while, as the HTTP response won't end with an EOF.

Upvotes: 0

bvr
bvr

Reputation: 9697

LWP::UserAgent dies with the error you are getting if there is no schema specified in request. It probably need it to properly work with it.

So, to make it work, you need to specify full url for your request:

my $req_str = "GET http://www.example.de/\n".
  "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1\n".
  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\n".
  "Accept-Language: en-us,en;q=0.5\n".
  "Accept-Encoding: gzip, deflate\n".
  "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\n".
  "Keep-Alive: 115\n".
  "Connection: keep-alive\n";

Upvotes: 1

Related Questions