How to parse bad html?

Question

I am writing a search engine that goes to all my company affiliates websites parse html and stores them in database. These websites are really old and are not html compliant out of 100000 websites around 25% have bad html that makes it difficult to parse. I need to write a c# code that might fix bad html and then parse the contents or come up with a solution that will address above said issue. If you are sitting on idea, an actual hint or code snippet would help.

Answer 1

Use a tagsoup parser, I'm sure the is one for C#. Then you can serialize the DOM to a more-or less valid HTML, depending on whether that parser conforms to the HTML DTD. Alternatively you can use HTML Tidy, which will clear at least the worst faults.

Regexes are not applicable for this task.

Answer 2

People generally use some form of heuristic-driven tag soup parser.

E.g. for

• Java
• Haskell

These are mostly just lexers, that try their best to build an AST from all the random symbols.

Answer 3

Just use Html Agility Pack. It is the very good to parse faulty html code