CODEWITHSUNDEEP
perlweb-crawlerlibwww-perl
Nagaraju
Nagaraju

Reputation: 1875

Web Crawler issue in perl

I had built a web crawler in Perl.

I am using 

HTML::ContentExtractor

LWP::UserAgent 

HTML::LinkExtor

to extract text form webpages.

Reference link for sample code web cralwer perl

Issue:

The issue is that it does not get text from web pages that have the extension as .aspx . It works perfectly for other webpages.I could not figure out the issue why this crawler fails for aspx pages.

Upvotes: 0

Views: 95

Answers (1)

Miller
Miller

Reputation: 35198

To be able to access a process a website with JavaScript, use WWW::Mechanize::Firefox

Upvotes: 1

Related Questions