Reputation:
Httrack can download all the files stored on the server of any website. How can I stop httrack to do so? can I achieve this by using robot.txt file ?
Upvotes: -1
Views: 1035
Reputation: 11
This can be done in two ways as follows:
1st Method:
User-agent: HTTrack
Disallow: /
2nd Method:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^HTTrack
RewriteRule ^.* - [F,L]
Upvotes: 0