CODEWITHSUNDEEP
html-parsingweb-crawlertext-mining
user68142
user68142

Reputation: 21

Extracting meaning full content from web pages

I am doing some analysis by mining web content using my crawlers. Web pages often contain clutter (such as ads, unnecessary images and extraneous links) around the body of an article that distracts a user from actual content.

To extract the sensible content is a difficult problem as I understand it, considering the fact that there is no standard that defines the actual position of a news-story/blog post/forum comment/article in the web page.

I could find some open source solutions like this: https://metacpan.org/pod/HTML::ContentExtractor

But I am curious if anyone has dealt with this and got reasonable success rate. It seems a fairly common problem and I would like to believe many experts are out there. I would prefer a JAVA based solution but that is not a hard rule. Please give some inputs. I will deeply appreciate.

Upvotes: 2

Views: 943

Answers (2)

joeytwiddle
joeytwiddle

Reputation: 31285

There are now a number of projects with this task as their primary goal.

The NPM package WCE (Javascript) is interesting because it uses a number of other content-extraction modules under the hood.

Sorry I meant to reply to this question earlier but I was busy.

Upvotes: 0

Andrew Vit
Andrew Vit

Reputation: 19249

Ideally, you would look for an RSS feed to get the raw content.

The is no standard for overall structure & meaning in HTML. Authors define different elements in their page. Search engines have invested a lot into this area, and they have their own secret sauce for indexing the content and getting some kind of meaning & structure out of it for search ranking.

Until we have the long-foretold "semantic web", we can only make educated guesses about the structure and meaning of arbitrary HTML pages.

But, in theory:

Look for heading tags. These should give you a clue for where to start reading, and hopefully an outline for the order of importance for the content.

Look for common element id and classes. A well-structured site might have things like <div id="content"> and <div class="article">, which is as semantic as it gets these days. Also get to know the standard element names used by common CMS platforms like WordPress ("post") or Drupal ("node"). Often these will be used to mark up the content.

Last but not least, look for microformats.

Upvotes: 1

Related Questions