Issue Server Health wrong stats

[Kapnos]

Hi,

from the beginning using Plesk Onyx on a fresh installation on a dedicated server I see that apache usage is always High but on apache status page (mod_status) and php status page I don't see anything that can explain this.


Apache Status

Server Version: Apache/2.4.10 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.0.1f
Server MPM: event
Server Built: Aug 31 2016 15:54:08
Current Time: Monday, 30-Jan-2017 10:30:24 CET
Restart Time: Sunday, 29-Jan-2017 11:21:37 CET
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 23 hours 8 minutes 47 seconds
Server load: 0.98 0.95 0.91
Total accesses: 824649 - Total Traffic: 5.6 GB
CPU Usage: u9.27 s2.73 cu0 cs0 - .0144% CPU load
9.9 requests/sec - 70.7 kB/second - 7.1 kB/request
3 requests currently being processed, 122 idle workers
PID    Connections    Threads    Async connections
total    accepting    busy    idle    writing    keep-alive    closing
23217    1    yes    2    23    0    0    0
23189    0    yes    0    25    0    0    0
23253    1    yes    1    24    0    0    0
23680    0    yes    0    25    0    0    0
23289    0    yes    0    25    0    0    0
Sum    2         3    122    0    0    0
______W_______________W_________________________________________
______W______________________________________________________...
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
........................................

PHP-FPM Status

pool:                 domain.tld
process manager:      ondemand
start time:           28/Jan/2017:11:12:37 +0100
start since:          170320
accepted conn:        347701
listen queue:         0
max listen queue:     0
listen queue len:     0
idle processes:       9
active processes:     2
total processes:      11
max active processes: 26
max children reached: 0
slow requests:        0

 

[Bitpalast]

Steps to tackle this issue:
1) # ps aux | grep httpd
Identify the httpd processes that are causing the high CPU load.
2) strace -p PID
with PID = the process id of these processes
3) From the output of strace identify the website that is doing transactions. Then look at the website's access_log and error_log entries to identify the script that is causing the issue.

 

[Kapnos]

None of the processes have high cpu load

root     22332  0.0  0.0 124340  8420 ?        Ss   Jan29   0:03 /usr/sbin/apache2 -k start
www-data 22335  0.0  0.0 121516  4280 ?        S    Jan29   0:00 /usr/sbin/apache2 -k start
www-data 22336  0.0  0.0 122692  4348 ?        S    Jan29   0:01 /usr/sbin/apache2 -k start
www-data 29242  0.5  0.0 2053776 15088 ?       Sl   19:24   0:01 /usr/sbin/apache2 -k start
www-data 29376  0.5  0.0 2053244 14224 ?       Sl   19:26   0:00 /usr/sbin/apache2 -k start
www-data 29425  0.4  0.0 2053528 12920 ?       Sl   19:27   0:00 /usr/sbin/apache2 -k start
www-data 29479  0.5  0.0 2052716 13336 ?       Sl   19:27   0:00 /usr/sbin/apache2 -k start
www-data 29530  0.6  0.0 2053532 13404 ?       Sl   19:28   0:00 /usr/sbin/apache2 -k start

 

[Bitpalast]

In
# top
command, what are the processes that cause the highest CPU load?

 

[Kapnos]

Usually mysql like this

top - 19:41:53 up 17 days, 23:49,  1 user,  load average: 0.46, 0.69, 0.88
Tasks: 195 total,   3 running, 192 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.5 us,  1.0 sy,  0.0 ni, 92.8 id,  0.6 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:  32727836 total, 24715752 used,  8012084 free,   809176 buffers
KiB Swap: 25770996 total,   169228 used, 25601768 free. 13748196 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
27985 mysql     20   0 12.759g 5.690g   7012 S  15.6 18.2   2873:31 mysqld
30090 thessal+  20   0  502388  77884  47132 S  12.3  0.2   0:08.13 php-fpm
29237 thessal+  20   0  516928  93584  50288 S   9.6  0.3   0:32.97 php-fpm
29945 thessal+  20   0  503268  78596  48912 S   9.3  0.2   0:12.38 php-fpm
 2885 root      20   0 4244080 1.123g   1048 S   3.0  3.6 322:47.56 redis-serv+
30317 www-data  20   0 2054196  14556   2708 S   0.7  0.0   0:00.99 apache2
30499 www-data  20   0 2053272  12072   2452 S   0.7  0.0   0:00.57 apache2
    7 root      20   0       0      0      0 R   0.3  0.0  39:28.34 rcu_sched
    8 root      20   0       0      0      0 S   0.3  0.0  28:15.69 rcuos/0
   13 root      20   0       0      0      0 S   0.3  0.0  14:20.77 rcuos/5
  102 root      20   0       0      0      0 S   0.3  0.0   3:59.98 kworker/7:1
  236 root      20   0       0      0      0 S   0.3  0.0 100:36.89 md1_raid5
  255 root      20   0       0      0      0 S   0.3  0.0  10:59.00 jbd2/md1-8
22601 nginx     20   0   37784  13180   2272 S   0.3  0.0  18:41.87 nginx
30354 www-data  20   0 2053304  12912   2452 S   0.3  0.0   0:00.62 apache2
30386 www-data  20   0 2053372  13492   2512 S   0.3  0.0   0:00.69 apache2
30620 ethessa+  20   0   24900   1688   1168 R   0.3  0.0   0:00.03 top

 

[Bitpalast]

This means that the user account thessal... is using a third of all CPU power, probably even more, because a major part of the mysql CPU usage will be associated with that account (as others don't show up at all).

I suggest to first investigate in-depth what that user is doing. I'd look at the access_log and error_log files of that account, and I'd also check the processes of mysql (login to # plesk db, then enter: show processlist.

 

[Kapnos]

Here is the mysql processlist

+---------+--------+-----------+--------+---------+------+----------------------+------------------------------------------------------------------------------------------------------+
| Id      | User   | Host      | db     | Command | Time | State                | Info                                                                                                 |
+---------+--------+-----------+--------+---------+------+----------------------+------------------------------------------------------------------------------------------------------+
| 1110142 | admin  | localhost | psa    | Sleep   | 8211 |                      | NULL                                                                                                 |
| 1232897 | admin  | localhost | psa    | Sleep   |  282 |                      | NULL                                                                                                 |
| 1233106 | admin  | localhost | psa    | Query   |    0 | NULL                 | show processlist                                                                                     |
| 1233440 | thesdb | localhost | thesdb | Sleep   |    1 |                      | NULL                                                                                                 |
| 1233441 | thesdb | localhost | thesdb | Query   |    0 | Copying to tmp table | SELECT SQL_CALC_FOUND_ROWS  wp_posts.ID FROM wp_posts  LEFT JOIN wp_term_relationships ON (wp_posts. |
+---------+--------+-----------+--------+---------+------+----------------------+------------------------------------------------------------------------------------------------------+

just some regular wordpress queries

and here are some entries on the error log that are interesting, they appear a lot:

AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace., referer: https://domain.tld/

AH01071: Got error 'PHP message: PHP Warning: fopen(https://i1.wp.com/domain.tld/wp-content/uploads/2013/01/\xce\xa3\xce\x95\xce\x9b-15-\xcf\x861-\xce\x94\xce\x99\xce\xa3\xce\xa4\xce\x97\xce\x9b\xce\x97.jpg?resize=500%2C335&ssl=1): failed to open stream: HTTP request failed! HTTP/1.1 400 Bad Request\r\n in /var/www/vhosts/domain.tld/httpdocs/wp-content/plugins/amp/includes/lib/class-fastimage.php on line 31\n'

After some digging I deleted some Rewrite rules in .htaccess and disabled the AMP plugin. Now I don't get any errors in apache logs but the CPU Usage remains high.

 

[Bitpalast]

If it is not for the errors, it is for access. We have seen similar situations here where customers were using asynchronous scripts or counter plugins, that were POSTing and GETing data once a second.

 

[Kapnos]

Nope, nothing strange in the error logs, casual GET requests of the website's resources (html, graphics and cached content)

 

[Bitpalast]

I don't think that is possible. The process list clearly states that the load is caused by PHP scripts run by that user. These scripts must be mentioned in the access_log.

 

[Kapnos]

You can check the logs here: https://dl.dropboxusercontent.com/u/1027498/logs.zip

 

[Bitpalast]

In my opinion there are several reasons for that the website is causing a high load. Here are a few of them and a suggestion how to solve them:

Issue: Very frequent downloads by Googlebot.
Suggested solution: Add website to Google Webmaster Tools, set a limit to Googlebot how frequent the website shall be spidered.

Issue: Almost all files including static resources create a 301 redirect, leading to at least one additional request per request (doubling the number of requests).
Suggested solution: Check if all these redirects are really necessary, e.g. either update rewrite rules in your .htaccess file or disable SSL enforcement (allow files to be downloaded unencrypted, too).

Issue: Static resources processed by Apache, not Nginx.
Suggested solution: Make sure that static files are delivered by Nginx (use Nginx as reverse proxy and let it deliver static resources like images directly). This will definitely remove a lot of unnecessary load, because Nginx will cache the contents and uses much less system resources in general. Currently you can see a wealth of .jpg files delivered through Apa.

Issue: Frequent errors on image processing through a script. "Got error 'PHP message: Aq_Resize.process() error: Image file does not exist (or is not an image): ..."
Suggested solution: Errors are always a burden on the server, because each error means that the server started something, then failed, then has to write the error message in addition to the time it already used for the process. Make sure these errors are resolved. On-the-fly image processing is a bad idea, anyway.