CODEWITHSUNDEEP
nginxuser-agent
VSOP_juDGe
VSOP_juDGe

Reputation: 51

Nginx don't block bot by user-agent

I'm trying to ban annoying bot by user-agent. I put this into server section of nginx config:

server {

    listen 80 default_server;

    ....

    if ($http_user_agent ~* (AhrefsBot)) {
       return 444;
    }

checking by curl:

    [root@vm85559 site_avaliable]# curl -I -H 'User-agent: Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)' localhost/
curl: (52) Empty reply from server

so i check /var/log/nginx/access.log and i see some connections get 444, but another connections get 200!

51.255.65.78 - - [25/Jun/2017:15:47:36 +0300 - -] "GET /product/kovriki-avtomobilnie/volkswagen/?PAGEN_1=10 HTTP/1.1" 444 0 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" 1498394856.155
217.182.132.60 - - [25/Jun/2017:15:47:50 +0300 - 2.301] "GET /product/bryzgoviki/toyota/ HTTP/1.1" 200 14500 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" 1498394870.955

How is it possible?

Upvotes: 0

Views: 1931

Answers (1)

VSOP_juDGe
VSOP_juDGe

Reputation: 51

Ok, got it! I've add $server_name and $server_addr to nginx log format, and saw that cunning bot connects by ip without server_name:

51.255.65.40 - _ *myip* - [25/Jun/2017:16:22:27 +0300 - 2.449] "GET /product/soyuz_96_2/mitsubishi/l200/ HTTP/1.1" 200 9974 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" 1498396947.308

so i added this and bot can't connect anymore

server {
            listen *myip*:80;
            server_name _;
            return 403;
        }

Upvotes: 1

Related Questions