CODEWITHSUNDEEP
htmlparsinghtml-parsingtext-processing
vent
vent

Reputation: 1063

Parsing Random Web Pages

I need to parse a bunch of random pages and add them to a DB. I am thinking of using regular expressions but I was wondering if there are any 'special' techniques (other than looking for content between known text/tags). The content is more(not always) like: 

Some Title
Text related to Title

I guess I don't need to extract complete Text but some way to know where the Title/Paragraph and extract the content from there. The content itself may have images/links that I would like to retain.

Thanks!

Upvotes: 0

Views: 159

Answers (3)

Daniel Cassidy
Daniel Cassidy

Reputation: 25587

You need to use a proper HTML parser, and extract the elements you’re interested in via the parser’s API (or via the DOM).

Since I don’t know what language you’re programming in, it’s rather difficult to recommend a parser, but some well known ones are Jericho for Java, and Beautiful Soup for Python.

Upvotes: 0

Daniel Cassidy
Daniel Cassidy

Reputation: 25587

Please see this answer: RegEx match open tags except XHTML self-contained tags

Upvotes: 1

S.Lott
S.Lott

Reputation: 391818

  1. Use Python. http://www.python.org/

  2. Use Beautiful Soup. http://www.crummy.com/software/BeautifulSoup/

Upvotes: 0

Related Questions