Question ByteSpider bots

[Inma]

Server operating system version

Debian

Plesk version and microupdate number

18.0.57

Hello

I don't care if I use the firewall or a fail 2ban jail or what .. but I need to block the how ByteSpider bots.

any ideas or suggestions?

 

[Peter Debik]

Please checkout the new How to Avoid High CPU Load & Block Bad Bots with Plesk article. It includes a Bytespider entry, but you'll also find lots of other useful information.

 

[Inma]

Thanks

 

[Inma]

thanks I tried it and nothing, I don't know what I have done wrong.
if you are so kind as to help me?


here you have the files folder etc/fail2ban/filter.d/apache-badbots.conf created with the content that you tell me...it is not....

is my file missing something?

 

[Peter Debik]

Could you please try to omit the "datepattern" lines (remove it from your file)? (The entry consists of two lines, remove both. so that the last valid line in the file is the "ignoreregex = ")

 

[JVG]

I usually update the apache-badbots fail2ban filter directly from the Plesk GUI, my entry for Bytespider looks like this:

Mozilla\/5\.0 \(Linux; Android 5\.0\).*\(compatible; Bytespider; spider-feedback@bytedance\.com\)

Added at the end of the badbots section.

I think it will be a great improvement for Plesk if the GUI was more friendly to the fail2ban configuration.

 

[Inma]

when I launch the test it stays like this... I understand that I have to wait and if it works I will have blocked IPs in the fail2 ban... I understandt

Thanks

 

[Peter Debik]

when I launch the test it stays like this...

If your log file is large or your computing power is slow, it can take a while to evaluate the file.

 

[Inma]

and from when it bans ips or prevents bots from accessing ? because they continue to access ... and the jail does not block anything

I keep getting bots logs that makes me high cpu consumption.

I don't know what else to do

 

[Peter Debik]

Did you check with fail2ban-regex against the access_ssl_log file of your domain that you achieve hits? If you do not see any matches, the rules need to be refined. I suggest to post a typical "Bytespider" log entry here, maybe we can find out together why it is not matched. Most often this happens when the bot name is enclosed in brackets or followed by a slash and a version number, then a refined rule could be something like \(Bytespider/, but it depends strongly on the string that is actually shown in the log.

 

[Inma]

the console closes before I can see the result of the log, for example. (domain.com is invented here is the real domain, but I have changed it so you can't see it.)

--


47.128.99.106 - - [15/Feb/2024:15:56:19 +0100] "GET /img/cms/pagos.png HTTP/1.0" 200 761 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-S-XL-U-39\/42-44\/47-43\/47/Marca-GIST" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /15872-home_default/calcetines-castelli-endurance-.jpg HTTP/1.0" 200 4290 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /14645-home_default/calcetines-endura-jagged.jpg HTTP/1.0" 200 6371 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /9886-home_default/calcetines-invierno.jpg HTTP/1.0" 200 8907 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"
47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /10715-home_default/calcetines-castelli-gdi.jpg HTTP/1.0" 200 5865 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /22435-home_default/calcetines-endura-hummvee-wtp.jpg HTTP/1.0" 200 6013 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /15012-home_default/calcetines-castelli-compressione-.jpg HTTP/1.0" 200 4956 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /16020-home_default/calcetines-castelli-premio-.jpg HTTP/1.0" 200 4747 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /13414-home_default/calcetin-northwave-eat-my-dust.jpg HTTP/1.0" 200 8334 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /15010-home_default/calcetines-castelli-compressione-.jpg HTTP/1.0" 200 3681 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /9885-home_default/calcetines-invierno.jpg HTTP/1.0" 200 5915 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /14972-home_default/calcetines-assos-duathlon-s.jpg HTTP/1.0" 200 3714 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /15876-home_default/calcetines-castelli-endurance-.jpg HTTP/1.0" 200 4968 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

47.128.43.215 - - [15/Feb/2024:15:56:19 +0100] "GET /9869-home_default/calcetines-lana-merino.jpg HTTP/1.0" 200 5677 "https://www.dominio.com/3066-calcetines-hombre?q=Talla-L-LXL-L\/XL-L\-XL-40\/43" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])"

66.249.64.102 - - [15/Feb/2024:15:56:19 +0100] "GET /bicicleta-gravel-allroad/9090-14802-basso-palta-ii-grx-600-2x11--8055136081579.html HTTP/1.0" 200 34589 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.6167.139 Mobile Safari/537.36 (compatible; Googlebot/2.1; +What Is Googlebot | Google Search Central | Documentation | Google for Developers)"

 

[Inma]

I usually update the apache-badbots fail2ban filter directly from the Plesk GUI, my entry for Bytespider looks like this:

Mozilla\/5\.0 \(Linux; Android 5\.0\).*\(compatible; Bytespider; spider-feedback@bytedance\.com\)

Added at the end of the badbots section.

I think it will be a great improvement for Plesk if the GUI was more friendly to the fail2ban configuration.

Here?

 

[Peter Debik]

As it is "Bytespider;" I suggest including the colon ";" in your apache-badbots.conf. Look, you will need to understand what the regular expression does. You could extend it for cases where the bot name is not followed by a space or quotation mark or whatever. There is no one size fits all solution for everything. You could easily play around a bit with the bot name, test it with fail2ban-regex until you see the matches.

 

[JVG]

Here?

View attachment 25516

No, that's the jail, you have to go to "Jails->Manage Filters":

and then select the "apache-badbots" filter:

 

[JVG]

Also, as @Peter Debik notes, these regular expressions need to be refined for each case, and might need to get updated and refined over time, as you could see, mine is maybe too constrained to the exact text of what my server was receiving at one point.

I use this site to test the regular expressions with the actual entries of the access log: RegExr: Learn, Build, & Test RegEx

 

[Inma]

No, that's the jail, you have to go to "Jails->Manage Filters":

and then select the "apache-badbots" filter:

View attachment 25519

It does not accept it, is this how you put it? pass me a wider capture please... I would like to do it from plesk, not from ssh... and there is no way.

Once I accept it I will see the rules and that but first I accept it.

Thanks

 

[JVG]

It does not accept it, is this how you put it? pass me a wider capture please... I would like to do it from plesk, not from ssh... and there is no way.

Once I accept it I will see the rules and that but first I accept it.

Thanks


View attachment 25521

Hi, you're again editing the plesk-apache-badbot jail. You need to edit the apache-badbots filter. In the Jails tab you have the "Manage Filters" button, that's where you need to go to edit the apache-badbots filter.

The plesk-apache-badbot jail should remain unchanged.

 

[Peter Debik]

@Inma , please review the blog article. It explains the difference of the /etc/fail2ban/jail.local and the /etc/fail2ban/filter.d/apache-badbots.conf file. You need to edit the /etc/fail2ban/filter.d/apache-badbots.conf file.

 

[Inma]

Ok, Gracias