Wordpress Toolkit - Enable Bot Protection why python-requests?

[ip00]

Server operating system version

Ubuntu 24.04

Plesk version and microupdate number

18.0.76 Update 1

The bot protection rule lists "python-requests" in its user agent blocklist. This is far too broad as python-requests is a generic Python HTTP library used by countless legitimate applications: open data portals (CKAN), monitoring tools, data integrations, government services, etc.
In our case, organization uses CKAN with python-requests to fetch public data from our API, and it was silently getting 403 responses. The endpoint has permission_callback => __return_true — it's intentionally public — but bot protection was blocking it at the Nginx level.

The other entries in the blocklist (acunetix, nikto, AhrefsBot, SemrushBot, etc.) are specific bad actors and fair game. But python-requests is equivalent to blocking curl or wget — you're blocking the tool, not a specific bad actor.

I think, Plesk should remove python-requests from this list. A legitimate bot protection rule should target known malicious user agents, not generic HTTP libraries.

Current list:

acunetix
BLEXBot
domaincrawler.com
LinkpadBot
MJ12bot/v
majestic12.co.uk
AhrefsBot
TwengaBot
SemrushBot
nikto
winhttp
Xenu Link Sleuth
Baiduspider
HTTrack
clshttp
harvest
extract
grab
miner
python-requests

 

[Sebahat.hadzhi]

Thank you for the feedback, @ip00 . python-requests is included in the list because many bots, automated scanners, and scrapers use the default Python requests User-Agent. So, it is an easy way to stop automated abuse without affecting real visitors. It can potentially cause issues if you have a legitimate Python script calling your site without setting a custom User-Agent.

 

[ip00]

@Sebahat.hadzhi The issue is that python-requests is a generic library, not a specific bot. Following this logic, Plesk should also block curl, wget, and Go-http-client, as they are also used by 'many bots and scanners'.

By blocking the default library UA, you aren't stopping sophisticated attackers (who spoof Chrome UAs anyway); you are primarily breaking out-of-the-box compatibility for legitimate open-source integrations like CKAN - CKAN - The open source data management system and automated government data feeds, or Dataverse, Sentry, UptimeRobot, Home Assistant, Apprise...

Generic tools should not be in a blacklist alongside known malicious signatures like acunetix or nikto. I suggest moving generic libraries to an 'Optional/Aggressive' list rather than the default protection."