How to discover common information blocks from multiple web pages of a same website?

Question

It is a pattern recognition task in web crawler. The traditional crawler gets the data of the whole page. If there is any way to make the crawler a litter intelligence, like just to identify and capture the the information part.

Answer 1

It is a research problem called wrapper induction or web data extraction. I don't know any library for this, but there are a lot of research papers (see below the list of good ones IMHO) and some research projects like DIADEM (their site contains list of publications as well).

• Muslea, Ion, Steven Minton, and Craig A. Knoblock. “Hierarchical Wrapper Induction for Semistructured Information Sources.” Autonomous Agents and Multi-Agent Systems 4, no. 1–2 (2001): 93–114.
• Dalvi, Nilesh, Ravi Kumar, and Mohamed Soliman. “Automatic Wrappers for Large Scale Web Extraction.” Proceedings of the VLDB Endowment 4, no. 4 (2011): 219–230.
• Dalvi, Nilesh, Ashwin Machanavajjhala, and Bo Pang. “An Analysis of Structured Data on the Web.” Proceedings of the VLDB Endowment 5, no. 7 (2012): 680–691.
• Gentile, Anna Lisa, Ziqi Zhang, Isabelle Augenstein, and Fabio Ciravegna. “Unsupervised Wrapper Induction Using Linked Data.” In Proceedings of the Seventh International Conference on Knowledge Capture, 41–48, 2013.
• Weninger, Tim, and Jiawei Han. “Exploring Structure and Content on the Web: Extraction and Integration of the Semi-Structured Web.” In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, 779–780, 2013. http://dl.acm.org/citation.cfm?id=2433499.