Question Best way to block bad user agents in global Nginx configuration?

[goomba]

Server operating system version

CentOS 7

Plesk version and microupdate number

18.0.58

I have tried the following in a file like /etc/nginx/conf.d/bad_bots.conf, but it still allows successful HTTP response.

map $http_user_agent $bad_user_agent {
    default             0;
    ~*MJ12bot           1;
    ~*coccocbot-image   1;
    ~*Baiduspider       1;
}

server {
    if ($bad_user_agent) {
        return 403;
    }
}

I've been able to successfully block them for individual domains through the Plesk interface with an if ($http_user_agent ... ) approach. How can this be done globally? Also open to a global Apache method.

 

[Peter Debik]

You'd rather want a rule lik

if ($http_user_agent ~* "(?:acunetix|BLEXBot|domaincrawler\.com|LinkpadBot|MJ12bot/v|majestic12\.co\.uk|AhrefsBot|TwengaBot|SemrushBot|nikto|winhttp|Xenu\s+Link\s+Sleuth|Baiduspider|HTTrack|clshttp|harvest|extract|grab|miner|python-requests)") {
    return 403;
}

However, why mess with Nginx if you have Fail2ban. I recommend reading this new article where we address the issue of bad bots and a lot more:

How to Avoid High CPU Load & Block Bad Bots with Plesk - Plesk

Learn how to effectively reduce CPU usage and keep bad bots and hackers away from your websites using Plesk Cgroups Manager and Fail2Ban.

www.plesk.com

 

[nisamudeen97]

Hi Peter,

I have tried adding the above rules server side to nginx include meanwhile i got the below error. Do you have some work around for the same?

nginx: [emerg] "if" directive is not allowed here in /etc/nginx/conf.d/bad_bot_block.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

 

[Kaspar@Plesk]

@nisamudeen97 have added the above ngnix directive to the Additional Nginx Directives in the nginx settings of the domain?