CODEWITHSUNDEEP
wget
DatsunBing
DatsunBing

Reputation: 9076

How to refuse wget?

I am uploading images to a public directory and I would like to prevent users from downloading the whole lot using wget. Is there a way to do this?

As far as I can see, there must be. I have found a number of sites where, as a public browser, I can download a single image, but as soon as I run wget against them I get a 403 (Forbidden). I have tried using the no-robot argument, but I'm still not able to download them. (I won't name the sites here, for security reasons).

Upvotes: 0

Views: 295

Answers (1)

permeakra
permeakra

Reputation: 622

You can restrict access using user-agent string, see apache 2.4 mod_authz_core for example. Wget also respects robots.txt directives by default. This should repent any casual user.

However, a careful look into wget manual will let to bypass these restrictions. Wget also lets to add random delays between requests, so even advanced techniques based on access pattern analysis may be bypassed.

So the proper way is to mess with wget link/reference recognition engine. Namely, the content you want to keep unmirrored should be loaded dynamically using javascript and the urls must be encoded in a way that would require js code to decode. This would protect your content, but would require to manually provide unobfuscated version for web bots you want to index your site, such as google bot (and no, it is not the only one one should care about). Also, some people do not run js scripts by default (esoteric browsers, low-end machines, mobile devices may demand such policy).

Upvotes: 2

Related Questions