Issue Unable to whitelist Cookiebot scanner in Plesk (tried all usual methods)

[Tim den Dikken]

Server operating system version

Ubuntu 24.04.3 LTS

Plesk version and microupdate number

Plesk Obsidian 18.0.72

Hi there fellow Pleskians,

I’m trying to whitelist the Cookiebot scanner in my Plesk server but can’t seem to get it working.
I’ve been going back and forward with their support and the result is that it’s really an issue on my side (I’ve also excluded it being a server-level problem as I tested the server plain without any firewall).

Cookiebot says their scans only ever come from the following static IPs:

20.223.9.138
34.107.102.47
34.141.10.24
34.159.86.126
34.159.168.195
34.159.247.222
35.198.78.207
35.198.137.6
35.198.160.49
35.246.143.2
35.246.191.14

…and their user agent looks like this:

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cookiebot/1.0; +Cookiebot™ CMP by Usercentrics - GDPR Compliance for SMBs) Chrome/X.X.X.X Safari/537.36

I’ve already tried all the obvious ways of allowing them but the scanner still gets blocked:

• Added the above IPs to Tools & Settings → Firewall (Plesk Firewall extension) as “allow incoming from” to test.
• Added them to Fail2Ban trusted IP list and disabled relevant jails temporarily to test.
• Added Allow from <IP> rules in .htaccess (Apache) and restarted Apache/Nginx
• Disabled ModSecurity temporarily to test and even tried adding a custom whitelist rule there

Despite all this, Cookiebot reports that it still can’t scan the site, the bot times out they say. Strange thing there, I have 0 logs of the bot ever trying to reach the server.
Which really makes you think it is a sever level issue, but I tried the server without Plesk and it works...

Has anyone successfully whitelisted the Cookiebot scanner in Plesk?
Do I maybe need to add something directly to the nginx.conf or Apache directives (like allow all with a specific user agent)?
Or is there a trick with Fail2Ban, mod_security rules, or Plesk’s Web Application Firewall that I’m missing?

Any tips or working configs would be greatly appreciated

 

[Kaspar]

Strange thing there, I have 0 logs of the bot ever trying to reach the server.

Which logs did you search trough?

 

[Tim den Dikken]

Which logs did you search trough?

I searched through all the logs available under Domain → Logs and enabled every option. Specifically, I reviewed the fail2ban.log and the modsec_audit.log.

 

[pleskpanel]

Do you run CSF? If you do, those IPs might be there and you wouldn't see them in Fail2Ban or the website logs.

 

[Tim den Dikken]

Do you run CSF? If you do, those IPs might be there and you wouldn't see them in Fail2Ban or the website logs.

That would have been a good guess! But sadly no... I never installed it, but checked anyways. It is not installed.

 

[Kaspar]

I searched through all the logs available under Domain → Logs and enabled every option. Specifically, I reviewed the fail2ban.log and the modsec_audit.log.

If there's no activity logged by fail2ban and mod_security you can pretty much rule those out. Another suggestion would be to manually analyze the iptables rules. See if there's anything in there that's blocking those IP's.

Are there any other applications installed/running that function as a firewall? Either as Plesk extension or on the server outside of Plesk. Maybe Imunify, CrowdSec, FirewallD? Or any proxy, like CloudFlare or your providers (cloud) firewall in front of your server that might filter traffic?

 

[Tim den Dikken]

If there's no activity logged by fail2ban and mod_security you can pretty much rule those out. Another suggestion would be to manually analyze the iptables rules. See if there's anything in there that's blocking those IP's.

Are there any other applications installed/running that function as a firewall? Either as Plesk extension or on the server outside of Plesk. Maybe Imunify, CrowdSec, FirewallD? Or any proxy, like CloudFlare or your providers (cloud) firewall in front of your server that might filter traffic?

Update:

I did some deeper checking to see if the Cookiebot IPs ever even reach the server as the tell me their scan times out, which is strange when it presumably would be blocked right...

Ran a live packet capture directly on the server:

sudo tcpdump -nn -i any \
\(host 20.223.9.138 or host 34.107.102.47 or host 34.141.10.24 or host 34.159.86.126 or host 34.159.168.195 or host 34.159.247.222 or host 35.198.78.207 or host 35.198.137.6 or host 35.198.160.49 or host 35.246.143.2 or host 35.246.191.14\) \
and \(port 80 or port 443\)

While Cookiebot tried to scan, no packets arrived from any of their IPs. That tells me the requests never hit the server’s network stack at all... so they’re being dropped or misrouted before they reach my server. It’s not Apache, .htaccess, WordPress, ModSecurity, or Fail2Ban blocking them; if the packets don’t appear in tcpdump, the server never saw them.

Their support keeps insisting they do, but I'm very certain they don't. As soon as I figure it out I will let you know of course! Bright ideas are very welcome, although I'm certain this isn't really Plesk related then....