CODEWITHSUNDEEP
jsonperlhtml
Karlyr
Karlyr

Reputation: 13

Saving a webpage for perl use

I'm trying to save a webpage during one of my perl scripts, but currently I can't find a way to properly save or get its content.

The web page content is basically a json file.

I looked for the chrome command line option "--save-page-as-mhtml" but I couldn't find a way to pass him the save location.

Upvotes: 1

Views: 256

Answers (2)

Sobrique
Sobrique

Reputation: 53498

At a very basic level you can get a web page like this using LWP:

# Create a user agent object
use LWP::UserAgent;
use HTTP::Request::Common qw( POST );

my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");

# Create a request
my $req = POST('http://search.cpan.org/search', [
   query => 'libwww-perl',
   mode  => 'dist',
]);

# Pass request to the user agent and get a response back
my $res = $ua->request($req);

# Check the outcome of the response
if ($res->is_success) {
    print $res->content;
}
else {
    die $res->status_line . "\n";
}

You can write $res -> content to a file easily enough, and that saves the JSON.

If it's JSON, you may find it useful to parse the JSON using the JSON library and potentially save the parsed JSON using Storable. (I'd generally suggest just saving the JSON as text and parsing it each time you load, but thought I'd offer Storable as it's quite a good way to turn an arbitrary perl data structure to an object on disk. )

Upvotes: 2

Dave Cross
Dave Cross

Reputation: 69314

If you just want a command line program to download and save a web page then look at wget, curl or lwp-request.

Upvotes: 1

Related Questions