Question Firewall rule to block bots

[SalvadorS]

Server operating system version

Debian 12

Plesk version and microupdate number

18.0.61

Hello Everybody,

We have a bot called claudebot who is hitting very hard a site. This bot has 1300+ different IPs, so my question is:

It is possible to block a bot in the plesk firewall by his name for example?

Something like if UserAgent contains claudebot then block.

Thank you in advance

 

[SalvadorS]

Or if you know how to block a bot in a plesk server for all websites let me know

 

[AYamshanov]

I think it can be done with fail2ban, it requires some customizations, see the next links for details

• How to Avoid High CPU Load & Block Bad Bots with Plesk
• fail2ban and openlitespeed · fail2ban fail2ban · Discussion #3745 (has an example of definition for "claudebot").

 

[Kaspar@Plesk]

I second @AYamshanov's suggestion. Fail2ban would be the right tool for this.

 

[SalvadorS]

Thank you. The articles are very interesting and I will test it.

Only one question, fail2ban an plesk firewall are compatibles?

 

[Kaspar@Plesk]

Only one question, fail2ban an plesk firewall are compatibles?

Yes

 

[Esperio]

Amazonbot is now in aggresive mode and it uses thousands ips to bypass fail2ban. In last 12 hours it has performed more than 200K request to one of my plesk servers.

 

[Esperio]

I came to a solution for this. I´ve created a gist explaining the steps to follow to block server wide all the ai training bots that are eating all resources from server:

Block AI Bots Server-wide in Plesk or ModSecurity-powered Servers

Block AI Bots Server-wide in Plesk or ModSecurity-powered Servers - block-ai-bots-server-wide.md

gist.github.com

Plesk: you can use it if you want, but please give credits

 

[AYamshanov]

Amazonbot is now in aggresive mode and it uses thousands ips to bypass fail2ban. [...]

What Plesk version do you use, and do you use fail2ban? In Plesk Obsidian 18.0.63, the fail2ban's bad bot list was updated, and the updated list also contains "AmazonBot",

 

[Cike76]

I use nginx for all my sites.
I have a rule for those Bad Bots

if ($http_user_agent ~* Paqlebot|Censys|Claudebot|serpstatbot|curl|Headless|ZyBorg) {

return 444;
}

444 response in nginx means nginx will throw away the connection ( not using anymore resources for that request )
That way ANYTHING that identifies as that bot in the user agent header will be blocked.

You will need to put this in the nginx rules section of the domains affected..

 

[pleskpanel]

I came to a solution for this. I´ve created a gist explaining the steps to follow to block server wide all the ai training bots that are eating all resources from server:

Block AI Bots Server-wide in Plesk or ModSecurity-powered Servers

Block AI Bots Server-wide in Plesk or ModSecurity-powered Servers - block-ai-bots-server-wide.md

gist.github.com

Plesk: you can use it if you want, but please give credits

This might be a bit more efficient, especially than using Fail2Ban which has to scan quite a few log files at scale (and there is a delay in looking at those). That script looks like it needs exact matches so it would be nice if it supported wildcard bot names.

 

[pleskpanel]

To follow-up on this thread, here is a quick way to take advantage of the built-in Plesk plesk-modsecurity jail which might handle requests a bit faster than scanning individual fail2ban regex/domain logs for bad bots user agent matches.

First, build a file in a path such as /mycustomdata/modsecurity/banned-user-agents.txt and add some banned user agents, one per line (keep in mind that this is both case insensitive and also finds any banned-user-agents.txt sub-string match in the browser's user agent). For example, a line in banned-user-agents.txt that contains "bad pleskybot" would match a user agent "A bad pleskybot 2.0" but not "bad plesky".

Now add this to your ModSecurity > Settings > Custom directives textarea:

# Block bots by User-Agent
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile /mycustomdata/modsecurity/banned-user-agents.txt" "phase:1,id:100002,deny,status:403,t:none,log,msg:'Found User-Agent associated with security scanner',logdata:'Matched Data: illegal User-Agent found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}'

This approach immediately drops the request (via deny), returns a 403 header to the requestor, and logs it.