Question Has my site been attacked?

[malware]

Server operating system version

AlmaLinux 8.8 (Sapphire Caracal)

Plesk version and microupdate number

Product: Plesk Obsidian 18.0.53 Update #2, last updated on June 21, 2023 03:44 AM

Dears,

my site is wordpress+woocommerce on dedicated server (Dell PowerEdge R250; Intel® Xeon® E-2324G 4x 3.10 GHz; 32 GB DDR4 - ECC).

This morning I couldn't open any page of my site from any computer/IP.
My browser gave me always ERR_CONNECTION_TIMED_OUT. E-mail connection wasn't possible, last e-mail update on my phone was at 3:34 am.

I wasn't at home so the quickly action was accessing to idrac and perform the power cycle system (cold boot). After that, the server and the site started working again.

Now, I want to understand what happened.

Here the error_log:

[Tue Jul 18 02:32:04.337043 2023] [fcgid:warn] [pid 331675:tid 140329894258432] [client 66.249.70.12:0] mod_fcgid: stderr: PHP Warning: Trying to access array offset on value of type bool in /var/www/vhosts/ribes.style/httpdocs/wp-content/plugins/woocommerce/includes/wc-template-functions.php on line 2673
[Tue Jul 18 02:39:02.233042 2023] [fcgid:warn] [pid 331675:tid 140330184771328] [client 66.249.70.12:0] mod_fcgid: stderr: PHP Warning: Trying to access array offset on value of type bool in /var/www/vhosts/ribes.style/httpdocs/wp-content/plugins/woocommerce/includes/wc-template-functions.php on line 2673
[Tue Jul 18 02:45:54.918928 2023] [fcgid:warn] [pid 299971:tid 140329458067200] [client 66.249.70.11:0] mod_fcgid: stderr: PHP Warning: Trying to access array offset on value of type bool in /var/www/vhosts/ribes.style/httpdocs/wp-content/plugins/woocommerce/includes/wc-template-functions.php on line 2673
[Tue Jul 18 02:59:48.543227 2023] [fcgid:warn] [pid 331675:tid 140329902651136] [client 66.249.70.13:0] mod_fcgid: stderr: PHP Warning: Trying to access array offset on value of type bool in /var/www/vhosts/ribes.style/httpdocs/wp-content/plugins/woocommerce/includes/wc-template-functions.php on line 2673
[Tue Jul 18 03:06:43.785580 2023] [fcgid:warn] [pid 299971:tid 140329441281792] [client 66.249.70.12:0] mod_fcgid: stderr: PHP Warning: Trying to access array offset on value of type bool in /var/www/vhosts/ribes.style/httpdocs/wp-content/plugins/woocommerce/includes/wc-template-functions.php on line 2673
[Tue Jul 18 03:37:51.952215 2023] [fcgid:warn] [pid 300188:tid 140328904410880] [client 66.249.70.11:0] mod_fcgid: stderr: PHP Warning: Trying to access array offset on value of type bool in /var/www/vhosts/ribes.style/httpdocs/wp-content/plugins/woocommerce/includes/wc-template-functions.php on line 2673

The IP is Google but there are few requests. After that there are only logs at 8am when I tried to open the site.

Here the access_log:

5.9.101.220 - - [18/Jul/2023:01:19:45 +0200] "GET /alfa-rex.php7 HTTP/1.1" 301 162 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:30:29 +0200] "GET /wp-content/plugins/ehjsu/ng.php HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:30:35 +0200] "GET /wp-content/plugins/fbajs/ng.php HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:31:50 +0200] "GET /wp-content/plugins/coba4/output/drunk.PHp HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:34:11 +0200] "GET /wp-content/plugins/okbtp/ng.php HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:34:32 +0200] "GET /wp-content/plugins/rpobm/ng.php HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:35:23 +0200] "GET /wp-content/plugins/coba5/output/drunk.PHp HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:35:30 +0200] "GET /wp-content/plugins/dsjoj/ng.php HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:35:50 +0200] "GET /wp-content/plugins/qllcp/ng.php HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"
94.102.208.129 - - [18/Jul/2023:02:35:56 +0200] "GET /wp-content/plugins/wp-file-explorer/output/drunk.PHp HTTP/1.1" 301 162 "www.ribes.style" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/587.38 (KHTML, like Gecko) Chrome/74.0.3729.169 Google/569 Safari/537.36"

This second log is anomalous, because I have not plugins that someone tried to open and the ip is reported on abuseipdb.

After that I have no recent logs.

Can you please suggest me where can I check further logs in order to understand what happened?

Thank you in advance

 

[MarkM]

error log is just warnings so not really a concern there.

you could have possibly triggered a temp IP block in your firewall, and by the time you rebooted enough time had lapsed.

pretty much shooting in the dark here though.

 

[malware]

Thank you.
I looked again into the issue, and I have found the following resources usage:


Can anyone please help me where can I find which service was using 27.3 GiB of memory-cached?

Thank you in advance

 

[malware]

Thank you.
I looked again into the issue, and I have found the following resources usage:
View attachment 23834

Can anyone please help me where can I find which service was using 27.3 GiB of memory-cached?

Thank you in advance

Sorry, I forgot to attach the cpu usage

 

[Peter Debik]

There is nothing wrong with RAM and cache. Linux has the habit to consume as much RAM as it can get.

Help! Linux ate my RAM!

The cpu does not look suspicious either. It is normal that after a server reboot the load goes up for a while, because starting services consume a lot of cpu power, especially web server and fail2ban.

 

[malware]

Thank you. Please note that cpu usage goes up before the server reboot, when the server was “ERR_CONNECTION_TIMED_OUT”

 

[Peter Debik]

Please check the log files of your domains where they show many requests. Normally a high load is caused by malicious bots that create many requests for a website. You can also check your process list (# ps aux) for processes that create a high load. Many times these are PHP-FPM processes, which again indicate a high load on a website due to frequent requests.

 

[jorge ceballos]

Hi,
Hopefully you solved it already but something similar was happening to my server.
It happened to be MariaDB eating all resources, though throughout the years it had never crashed the server, this past month started to happen every monday's noon. Load would rise over 90 and froze everything. Even rebooting the server took a while.
Using mysqltuner as a guide, configued MariaDB so the most resources it can use now is 85%.
That solved my issue and haven't crashed since.
Regards

 

[Peter Debik]

@jorge ceballos For your case it could also be beneficial to add the max_connections and max_user_connections variables into the "[mysqld]" section of /etc/my.cnf to limit the number of connections that a user account can create. That is, because some user accounts create persistent connections and build-up many connections by it, which uses more and more cpu resources from MariaDB. This eventually leads to a point where MariaDB does not have any time left to serve other requests and appears to crash. So limiting the total number of connections and the number of connections that a single user can do is an important step to increase the server's stability. typical values are max_user_connections=80 (very unlikely that in a correctly operating website more than 80 connections are needed at a time) and max_connections=1000 (depending on the cpu power of your system a bit more or a bit less).