Resolved User Agent Filtering

[CiretPro]

User Agent Filtering

Hello, tell me if there is a possibility of filtering traffic by specifying a user agent based on nginx.

Interested in server and client level solutions by specifying additional nginx directives

Example for server level:

if ($ http_user_agent ~ (ZmEu | libwww-perl | wget)) {
return 403;
}

I submit to etc/nginx.conf but does not work.

Thanks

 

[IgorG]

Try this one Instruction - Blocking extra bots using nginx

 

[CiretPro]

Thanks, today I’ll try to apply

 

[ChristophRo]

nginx does not support this on a global/server level and you have to add this configuration for every virtualhost.

So what you could do is:
1) create a file named /etc/nginx/baduagent.conf and put in what IgorG linked
2) create a custom template (/opt/psa/admin/conf/templates/custom/domain/nginxDomainVirtualHost.php) and add a "include baduagent.conf;" in an appropriate position
3) rebuild web configuration


You can also do it in Apache2, as here it's possible to configure it globally:

1) add file /etc/apache2/conf-enabled/baduagent.conf (this works for debian/ubuntu - for rpm based linux use similar/appropriate location) with the following content (add/remove/adjust UserAgents as needed)

# global blocking of bots we don't like
<IfModule setenvif_module>
  <Location />
        <If "%{HTTP_USER_AGENT} =~ /SemrushBot/">
                Require all denied
        </If>
        <If "%{HTTP_USER_AGENT} =~ /MJ12bot/">
                Require all denied
        </If>
        <If "%{HTTP_USER_AGENT} =~ /BLEXBot/">
                Require all denied
        </If>
        <If "%{HTTP_USER_AGENT} =~ /AhrefsBot/">
                Require all denied
        </If>
        <If "%{HTTP_USER_AGENT} =~ /DotBot/">
                Require all denied
        </If>
        <If "%{HTTP_USER_AGENT} =~ /XoviBot/">
                Require all denied
        </If>
        <If "%{HTTP_USER_AGENT} =~ /Seekport/">
                Require all denied
        </If>
  </Location>
</IfModule>

2) restart apache2 service




Advantage of nginx method is that it will save some (very few) resources, as requests do not need to go the Apache2 service first and it also works for websites where Apache2 is disabled completely. (i.e. nginx proxying is disabled)

Advantage of Apache2 method is that you do not need to fumble with the Plesk templates, because this became a pita with 18.0, as you need to check and possibly redo them on every Plesk update. (i.e. every couple days/weeks...)

 

[CiretPro]

Thanks!