CODEWITHSUNDEEP
perlhtml-parsing
Programmer
Programmer

Reputation: 8707

How can I extract the text from links on an HTML page in Perl?

I am not an expert in Perl but I have written a Perl script to parse an HTML page and filter by all href tags:

The output are as shown below:

href="?Name">Name</a>
href="?Desc">Hourly Details</a>
href="/24x7/2012/11-November/">Data
href="./00:00:00/">00:00:00/</a>
href="./01:00:00/">01:00:00/</a>
href="./02:00:00/">02:00:00/</a>
href="./03:00:00/">03:00:00/</a>
href="./04:00:00/">04:00:00/</a>
href="./05:00:00/">05:00:00/</a>
href="./06:00:00/">06:00:00/</a>
href="./07:00:00/">07:00:00/</a>
href="./08:00:00/">08:00:00/</a>
href="./09:00:00/">09:00:00/</a>
href="./10:00:00/">10:00:00/</a>
href="./11:00:00/">11:00:00/</a>
href="./12:00:00/">12:00:00/</a>
href="./13:00:00/">13:00:00/</a>
href="./14:00:00/">14:00:00/</a>
href="./15:00:00/">15:00:00/</a>
href="./16:00:00/">16:00:00/</a>
href="./17:00:00/">17:00:00/</a>
href="./18:00:00/">18:00:00/</a>
href="./19:00:00/">19:00:00/</a>
href="./20:00:00/">20:00:00/</a>
href="./21:00:00/">21:00:00/</a>
href="./22:00:00/">22:00:00/</a>
href="./23:00:00/">23:00:00/</a>

Now I want to extract values within the href tags from "00:00:00" till "23:00:00" while exclude others. The result value would be added to string having a URL:

http://x.download.com/00:00:00
------URL------------/..href../
..............................
http://x.download.com/23:00:00

However by trying the below code:

foreach (@tag) {
    if (m/href/) {
        if ($_ =~ /"\/24/ && $_ =~ /"\/[0-9]/) {
            my $href  = $_;
            my $start = index($href, "\"");
            my $end   = rindex($href, "\"");
            my $link  = substr($href, $start + 1, $end - $start - 1);
            print "Follow: " . $url . $link . "\n";

        }
    }
}

prints:

Follow: http://x.download.com/24x7/2012/11-November/

What should my regular expression be such that required objective can be achieved?

Upvotes: 1

Views: 612

Answers (3)

Axeman
Axeman

Reputation: 29854

First of all, we need to specify a regex which will capture military times to the second. 

my $regex 
    = qr{  # curly brackets instead of slashes
           # so that we can use literal slashes in expression

    "   # a quote
    \.  # a literal dot
    /   # a forward slash
    (   # begin capture group

       (?:              # begin uncaptured sub-group
           [01] \d      # a '0' or '1' followed by a digit
       |   2    [0-3]   # a '2' followed by 0-3
       )                # end grouping
       (?:         # begin repetition grouping
         :         # a literal colon               
         [0-5] \d  # digits 0-5 followed by any digit
       ){2}        # exactly twice
     )  # end capture

     /  # a forward slash
     "  # close quote
}x; # <- x-option allows annotated regex
...

This is equivalent to the following regex: 

my $regex = qr/"\.\/((?:[01]\d|2[0-3])(:[0-5]\d){2})\/"/;

If your minutes and seconds will only ever be '00:00', then the expression is even easier: 

my $regex = qr{"\./((?:[01]\d|2[0-3]):00:00)/"};

Then you can test and retrieve the value by making the match in a list context:

if ( my ( $link ) = m/$regex/ ) { 
    say "http://x.download.com/$link";
}

If the test does not match, $link will be undefined. If it does match, having declared it as a list (of one), the match operation will assign the first capture to the variable.

Upvotes: 0

Borodin
Borodin

Reputation: 126722

This is done very simply with a regular expression, as shown in the program below. It looks for a string of digits or colons immediately following > (and so looks for the text contents of the element rather the href attribute value as yours does) and captures that string into $1.

But I would prefer to see the problem solved from start to finish using a proper HTML parser, such as HTML::TreeBuilder or Mojo::DOM.

use strict;
use warnings;

my @tag = <DATA>;

foreach (@tag) {
  next unless />([\d:]+)/;
  print "http://x.download.com/$1\n";
}

__DATA__
href="?Name">Name</a>
href="?Desc">Hourly Details</a>
href="/24x7/2012/11-November/">Data
href="./00:00:00/">00:00:00/</a>
href="./01:00:00/">01:00:00/</a>
href="./02:00:00/">02:00:00/</a>
href="./03:00:00/">03:00:00/</a>
href="./04:00:00/">04:00:00/</a>
href="./05:00:00/">05:00:00/</a>
href="./06:00:00/">06:00:00/</a>
href="./07:00:00/">07:00:00/</a>
href="./08:00:00/">08:00:00/</a>
href="./09:00:00/">09:00:00/</a>
href="./10:00:00/">10:00:00/</a>

output

http://x.download.com/00:00:00
http://x.download.com/01:00:00
http://x.download.com/02:00:00
http://x.download.com/03:00:00
http://x.download.com/04:00:00
http://x.download.com/05:00:00
http://x.download.com/06:00:00
http://x.download.com/07:00:00
http://x.download.com/08:00:00
http://x.download.com/09:00:00
http://x.download.com/10:00:00

Upvotes: 3

Andy Lester
Andy Lester

Reputation: 93666

You do not want to do it with regular expressions. You need a proper HTML parser, and regexes cannot do the job.

How are you fetching the web page? If you're using WWW::Mechanize, then extracting the links from the page that you have fetched is a single method call, because WWW::Mechanize does the HTML parsing for you.

use WWW::Mechanize;

my $mech = WWW::Mechanize->new();
$mech->get( $url );

my @links = $mech->links();
for my $link ( @links ) {
    say $link->text, ' -> ', $link->url; # Show the text and the URL
}

You'll need to reformat as you see fit, but that gives you an idea.

Upvotes: 3

Related Questions