Issue php-fpm processes and mysql are maxing out server cpu

[Matthias Schleiermacher]

Server operating system version

Debian 10.13

Plesk version and microupdate number

18.0.59 #2

Hey friends.
Since 04:00 in the morning one of our servers started to have a very high load.
php-fpm pools and mysql ramp up to a serverload more than 200 ...
The only thing that helps is stopping mysql and killing php-fpm. Also we tried to stop different php versions, but nothing really helped.
Searched everywhere, all kind of logs, netstat and all that stuff...but nothing points us to the problem.
After digging the whole day we don't know what else to do, maybe some good advises?

 

[Kaspar@Plesk]

If I had to take a guess I would say your server (or rather one or more website on your server) are being attacked. Or some bots are flooding your websites. Based on the system user shown in the process list you should be able to determine which domain(s).

This link might help you: https://support.plesk.com/hc/en-us/...-find-websites-under-attack-on-a-Plesk-server

 

[Matthias Schleiermacher]

Thank you for your reply. Did that this morning and now again. And there is nothing suspicions.

 

[Matthias Schleiermacher]

A few hours everything was finde. since 06:30 it started again. Wenn we try
ss -tan state established | grep ":80\|:443" | awk '{print $4}'| cut -d':' -f1 | sort -n | uniq -c | sort -nr
the output is not much:

/var/www/vhosts/system # ss -tan state established | grep ":80\|:443" | awk '{print $4}'| cut -d':' -f1 | sort -n | uniq -c | sort -nr
      5 195.204.33.2
      3 92.221.64.26
      3 79.161.4.26
      3 212.33.158.58
      3 185.222.84.157
      3 [
      2 85.208.98.18
      2 45.140.0.100
      2 34.196.51.17
      2 213.0.108.66
      2 195.191.219.133
      2 195.191.219.132
      2 185.241.208.216
      2 127.0.0.1
      2 109.247.193.50
      1 97.74.211.129
      1 88.130.217.91
      1 84.165.144.222
      1 81.166.208.18
      1 81.166.126.41
      1 79.160.132.206
      1 77.16.217.174
      1 72.167.254.247
      1 69.162.124.231
      1 66.249.70.7
      1 62.92.155.205
      1 62.73.122.13
      1 52.21.236.0
      1 51.38.41.215
      1 51.222.253.14
      1 51.222.253.12
      1 51.195.216.60
      1 46.5.214.22
      1 45.135.241.228
      1 44.205.161.253
      1 37.157.255.193
      1 35.169.190.168
      1 3.139.50.99
      1 217.76.155.236
      1 213.157.22.140
      1 209.97.145.94
      1 208.109.71.255
      1 195.159.179.30
      1 194.5.175.226
      1 194.248.36.146
      1 192.42.116.178
      1 185.73.26.204
      1 18.216.127.212
      1 17.58.56.7
      1 165.232.180.139
      1 138.32.168.34
      1 137.184.98.241
      1 130.180.4.121
      1 103.108.45.105

Looks legit to me, Tried some lookup. The only strange is 3 [

 

[MartinT]

Look at plesk backend Domains -> Logs and filter statur 400. If I had to guess, some is trying sqlinjections. Then you have to work on your Sec-rules and or htaccces and/or proxy-dns like Cloudflare ... They do it by large caputred Cnetworks, so it looks like a single IP from the first view. With that behavior the ship around fail2ban (one Ip is only a single try, after the standard ban period the ip try again) Had someting like that last october.

 

[Bitpalast]

/var/www/vhosts/system # ss -tan state established | grep ":80\|:443" | awk '{print $4}'| cut -d':' -f1 | sort -n | uniq -c | sort -nr

Looking at current connections will probably not help. Instead, you'll need to check the access_ssl_log files of your website(s). A good start is to first check which users' PHP-FPM is active (like # ps aux | grep php-fpm). Normally you'll directly see where a lot of cpu load is created. Then check the website's access_ssl_log for frequent IP addresses other than your own, e.g. # cat access_ssl_log | awk '{print $1}' | sort -n | uniq -c | sort -n. Then either ban the culprit, e.g. # fail2ban-client set recidive banip <IP-address>. Or better: improve a suitable Fail2Ban jail definition to cover the case.

 

[Matthias Schleiermacher]

Thank you for your reply, i'll try your advise