CODEWITHSUNDEEP
apacheweb-crawler
Klaitos
Klaitos

Reputation: 137

Blocking web spider

I want to block all web spiders to vaccum my web site.

Is there a way ?

I only found some Apache rules from 2008 (like this one ) http://perishablepress.com/ultimate-htaccess-blacklist/

Upvotes: 1

Views: 429

Answers (1)

0x5C91
0x5C91

Reputation: 3515

Unfortunately, there isn't a way to block ALL scripts from accessing your site: if a human can, nothing prevents anyone from writing a spider that behaves similarly to a person and can therefore view every page. You can look up techniques to prevent certain robots from accessing your site (you can do it with the majority of search engines), but if it remains up long enough and gets some visits it's likely that it will eventually end up in some database.

Have a look here.

Upvotes: 2

Related Questions