CODEWITHSUNDEEP
pythondiffhtmlsnapshot
hoju
hoju

Reputation: 29472

how to determine if webpage has been modified

I have snapshots of multiple webpages taken at 2 times. What is a reliable method to determine which webpages have been modified?

I can't rely on something like an RSS feed, and I need to ignore minor noise like date text.

Ideally I am looking for a Python solution, but an intuitive algorithm would also be great.

Thanks!

Upvotes: 6

Views: 3030

Answers (4)

ghostdog74
ghostdog74

Reputation: 343087

just take snapshots of the files with MD5 or SHA1...if the values differ the next time you check, then they are modified.

Upvotes: -1

brianegge
brianegge

Reputation: 29892

The solution really depends if you are scraping a specific site, or are trying to create a program which will work for any site.

You can see which areas change frequently doing something like this:

 diff <(curl http://stackoverflow.com/questions/) <(sleep 15; curl http://stackoverflow.com/questions/)

If your only worried about a single site, you can create some sed expressions to filter out stuff like time stamps. You can repeat until no difference is shown for small fields.

The general problem is much harder, and I would suggest comparing the total word count on a page for starters.

Upvotes: 3

Luk&#225;š Lalinsk&#253;
Luk&#225;š Lalinsk&#253;

Reputation: 41316

Well, first you need to decide what is noise and what isn't. You can use a HTML parser like BeautifulSoup to remove the noise, pretty-print the result, and compare it as a string.

If you are looking for an automatic solution, you can use difflib.SequenceMatcher to calculate the differences between the pages, calculate the similarity and compare it to a threshold.

Upvotes: 8

Alex Sexton
Alex Sexton

Reputation: 10451

Something like Levenshtein Distance could come in handy if you set the threshold of the changes to a distance that ignored the right amount of noise for you.

Upvotes: 0

Related Questions