How do I crawl my own website?

Question

I've inherited an old Classic ASP website to modify. Although not requested up-front, I'd like to delete a bunch of the old "orphaned" pages.

For some reason, The old developer decided to create muliple instances of the file instead of using source control (eg. index-t.asp, index-feb09.asp, index-menutest.asp).

I'm wondering if anyone knows of a program or website, that can crawl my own site for me? It probably needs to be able to crawl public site, since there are lots of include files. Also, some of the urls are relative and some are absolute.

Answer 1

You should consider:

1. Putting the entire existing site into source control, then
2. Delete the extra pages and see who complains

Answer 2

You should never let a once-valid URL go stale. Bad web developer! No biscuit!!

Answer 3

There's also the W3C link checker: http://validator.w3.org/checklink

Answer 4

My favorite tool is Xenu.