taw
Reputation: 18861
How to quickly check large number of URLs if they're still valid (or 404/otherwise missing)
Urls are already conveniently in a text file, one line per url, so it's probably trivial curl/wget/LWP one-liner. Anybody cares to share such?
Upvotes: 0
Views: 248
Answers (1)
Vinko Vrsalovic
Reputation: 340286
With LWP you can do this (redirecting output to a file if you so wished)
linux-t77m$ cat urls
http://google.com
http://stackoverflow.com
http://yahoo.com
linux-t77m$ cat urls | while read i ;do echo -n $i" "; lwp-request $i -sd; done
http://google.com 200 OK
http://stackoverflow.com 200 OK
http://yahoo.com 200 OK
The quickest way (as in getting the results faster) would be to launch processes in parallel, of course.
Upvotes: 4
Related Questions
- Fastest way to check if a url exists
- Efficiently detect broken urls in Java
- Fast way to extract a list of URLs and check validity
- Checking 1000 urls if they exist or not, is there a quick way for this?
- How do I see if thousands of URLs are active without a timeout using C#?
- Automatic web tester for 404 links?
- Check a file of HTTP URLs for HTTP response codes
- How can I loop through a large list of URL's and determine if they are valid (non 404 response)?
- Checking a lot of URLs to see if they return 200. What's the cleverest way?
- What's the fastest way to test the validity of a large number of well-formed URLs