Single slow domain but everything else is fast

[icaravans]

Hey guys, I have been here a few times looking for advice and have always found it until now so I thought I would take the leap to register and ask a question about one of my servers.

I have a dedicated server running CentOS 6 and Plesk 12.5.30 and everything suggests that the server is not struggling in any way.

My Plesk panel is smooth, MySQL DBs are responsive, SSH is quick and emails are sent out in a timely manner.

What I find strange is that a single domain, admittedly a very busy one, is slow to respond to the point of timing out at peak times.

I have put timers in my php script that is the cause of the high traffic and it always finishes within 2 seconds but I guess connecting to the server at these peak times is the cause of the time outs.

I am assuming that there is a limit to the amount of concurrent connections allowed but I have tried FCGI, FPM, PHP 5 - 7, apache, nginx and also tried changing up a bunch of settings to no avail.

When the problem domain is timing out the rest of the server is still very responsive and the overall CPU usage rarely gets above a few percent in fact it often is below 1% CPU usage. Memory usage is never an issue and there is plenty of spare HDD space.

If you guys could give some pointers as to where I could look to improve this situation I would be very appreciative.

Thanks in advance guys.

 

[trialotto]

@icaravans

You could do a test in which the default memory allocation to the domain is increased: go to "Domains > [ domain ] > PHP Settings (click)" and change

- memory_limit to some considerable value: 256M or 512M could improve performance (do not set more, otherwise you will get into trouble)
- max_execution_time and max_input_time to: 120
- opcache.enable to: on (if it is not already enabled)

and note that it would be best to choose PHP 7 version.

Also, remove the timers from your php script, that also requires resources that actually should not be used (unless you are debugging and even then, better methods are available).

Finally, you should be more specific, in order to allow us to give you rock-solid advice.

Consider for instance the questions:

- are you having the problems on a WordPress instance? And if true, what kind of themes/plugins and how many plugins are used?
- what are the other settings for the domain? Did you do some customization (Nginx, PHP, etc.)?
- do you use many Apache modules and/or are many Apache modules activated?
- what is your Apache version (2.2 I suppose)?

and so on. Naturally, some output from Apache logs and other logs can also be helpful.

But before providing all the logs, first start to give us with some more details, after you have changed settings as described above.

Regards....

 

[icaravans]

Thanks for the reply:

memory_limit was set to 128MB so I would be surprised if that were the issue but I have increased it to 512MB to see how it goes, there is plenty of memory free on the machine so I could raise it up higher if this helps at all.
max times I set to 120 they were 60s execution and 30s input.
opcache was already set to on.

The time outs were being thrown after 60 secs of inactivity from the client yet when the pages were successfully returned they were usually completed within 1-2 seconds even at peak times so it seems like there is a long delay before starting the script rather than the script itself being what takes the time.

It is currently running on PHP 7 (latest version that would install). The timer was put in simply to check how long it takes for the script to get from the start to the end and never was it over a few seconds despite it taking many times longer to actually return.

There is no WP on the domain, just a static website with less than 100 page loads daily and the single script that is called by a great many users concurrently. The script has up to 3 curl requests per call but still completes within a max of a few seconds when it finally connects.

I have tried nginx and apache, many different installs of PHP v5 and v7 and FCGI and FPM, none of it really opened up the full resources of the server as this domain would still crawl to the point of timing out while the other domains and services are snappy with no sign of slow down whatsoever.
Apache v2.2.15
Apache Modules:

 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 substitute_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_ajp_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 bw_module (shared)
 fcgid_module (shared)
 perl_module (shared)
 php5_module (shared)
 python_module (shared)
 ssl_module (shared)
Syntax OK

I have attempted your changes and will see how it pans out, I hope it works out and will provide any further information needed.

Thanks.

 

[icaravans]

I realised that maybe this is what you meant by modules:

Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

Is this likely due to the MPM mode being prefork and not worker? After looking at it I see that it says "Threaded: No" which would explain why it appears that the pages are being returned sequentially

 

[trialotto]

@icaravans,

Apache 2.2 is a bit of an odd-ball in the Apache range and this is the disadvantage of having CentOS 6.

In general, with respect to the Apache mode, one can recommend the "prefork" mode (enable that, if you did not do that already) and note that "worker" mode is not an option.

However, "prefork" mode is still a form of multi-threading: one control process, multiple child processes handling requests.

Let´s not get in details to configure control and child processes, the chances are big that a big error can be the result of that.

In essence, it is the nature of the php script, with 3 curl requests: is there any way to change that?

After all, curl is not that "fast", since it is in "sync" mode by default: one needs to create a work-around to enable async curl requests.

There are some alternatives to allow for async curl requests, the most interesting are: sockets, forks or curl_multi.

I recommend to read up on the topic, there is enough (good and bad) documentation online.

In short, try to determine whether the new settings work a bit better AND whether the "slow script" can be improved.

As a final remark, it can do no harm to disable some Apache modules that you do not use.

Hope the above helps.

Regards

 

[icaravans]

It is already using prefork as far as I can tell.

The 3 curl requests are not avoidable in some cases. The results from them are cached such that subsequent requests for the same information is pulled from a MySQL DB but if the info is not yet cached then they need to be pulled via a curl request.

As I stated previously though I can see from the timers that even in the instances where the request takes upwards of 10-30s to return the script itself is going from start to finish within a second or 2 so the script running does not appear to be what is taking all of the time. I forgot to mention that the timer is only used when debugging so I can see exactly what is taking the most time and the curl requests are done to API servers that return results in less than 200ms in most cases.

What confuses me is when the script is getting hammered it can take up to 30s to return (or even timeout when really heavy load) yet the exact same script on a different domain on the same server will return within a second or two. This is what is confusing me, the script itself seems to be running fine, the server is not bogged down as everything is smooth but it seems that single domain is using up all of the resources that domain is allowed to despite me having removed as many of the limits as I can find for it.

I have tried multi curl in the past with no improvement to the speed.

I have not really been through the modules yet, maybe that is worth a bit of time to try out.

Thanks again for the help, it is appreciated and I will update with any changes in the way things are working out.

 

[trialotto]

@icaravans

Let´s await the result from the changes then, I will keep your last post in mind (and if something is coming to my mind, I will share it).

Regards......

 

[ugr|dual]

Sounds to me like your Apache2 Prefork section needs proper settings (tuning it for the purpose). Same for the MySQL server.

 

[icaravans]

Sounds to me like your Apache2 Prefork section needs proper settings (tuning it for the purpose). Same for the MySQL server.

Thanks. I have tried gradually increasing the maxclients and other settings in the prefork settings of the /etc/httpd/conf/httpd.conf file.

I have also increased the settings in the /etc/httpd/conf.d/fcgid.conf file which did allow for more connections per second but I still have the strange issue that when php on one domain is absolutely crippled to the point the script is timing out before it is even beginning the other domains and services are running along nice and smoothly.

I would understand the situation if the whole server was grinding to a halt but it is a single domain on the server with everything else being fine, literally everything else is fine (Plesk panel, other domains, MySQL) and loads within a fraction of a section whereas any PHP script on that domain will be timed out before it even gets to start.

 

[ugr|dual]

Here is an example of high load prefork apache2 basic settings:

Timeout 300
KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 5
# prefork MPM
<IfModule mpm_prefork_module>
    ServerLimit         500
    StartServers         10
    MinSpareServers      10
    MaxSpareServers      10
    MaxClients          500
    MaxRequestsPerChild   10000
</IfModule>

However, MySQL is a totally different thing and varies on the software you are running.

I recommend something like:

MySQL performance tuning primer script                #
#    Writen by: Matthew Montgomery

It works out the needs pretty good.

Let me know if you can't source that anymore.

As for PHP, as long as your memory settings are OK the bottleneck should not be there. But I use Apache-module and FPM-event only, can't say anything about cgi/fcgi limitations under high load.

 

[ugr|dual]

I have also increased the settings in the /etc/httpd/conf.d/fcgid.conf file which did allow for more connections per second but I still have the strange issue that when php on one domain is absolutely crippled to the point the script is timing out before it is even beginning the other domains and services are running along nice and smoothly. I would understand the situation if the whole server was grinding to a halt but it is a single domain on the server with everything else being fine, literally everything else is fine (Plesk panel, other domains, MySQL) and loads within a fraction of a section whereas any PHP script on that domain will be timed out before it even gets to start.

Well, if the software on that one domain is different this is very easily possible.

Some "simple" sites and one that is pulling huge results from MySQL with no proper indexing and lack of memory and therefor disc action.. there is your stuttering site while everything else keeps running (mostly) in memory.

 

[icaravans]

I'll look into the MySQL tuning but I use EMS SQL manager for MySQL and can happily do SELECTs on a DB with 3mil+ records without any sign of any real slow down so it wouldn't appear that the MySQL server or the SSD is bogged down. CPU usage is rarely at even 1% and memory is 2.6GB free of 7.8GB total.

The comparison between domains (slow domain against speedy domains) is done using the exact same script using the same resources and being sent the same $_GET variables.

The whole server is setup to run that one domain but I added a couple of spare domains I had to just test it.

I have setup the server-status script on the server to see what is going on and it really looks like when a bunch of concurrent requests are being made that they are just getting queued and only a few are being run at any one time, the main stats from that is below:

Current Time: Saturday, 09-Apr-2016 15:58:53 BST
Restart Time: Saturday, 09-Apr-2016 14:53:50 BST
Parent Server Generation: 0
Server uptime: 1 hour 5 minutes 3 seconds
Total accesses: 11608 - Total Traffic: 4.2 MB
CPU Usage: u3.68 s6.37 cu0 cs0 - .257% CPU load
2.97 requests/sec - 1134 B/second - 381 B/request
224 requests currently being processed, 90 idle workers

Note the CPU usage and the number of requests/sec

The time taken for the script to run is around 2-3 seconds avg with the occassional one hitting up to about 10s but some finishing in a small fraction of a second. Even still it is maxing at about 3 requests/sec. machine is an 8 core Atom with 8GB ram, SSD storage and 100Mb all dedicated to the machine.

I am fairly certain that the issue is with the setup of the server or the script itself but every test I have done on the script shows it finishing within a few seconds even when it takes 30s to be returned.

(example) So what is happening is that I request the script and it gets returned in 30s, I go through the log output and see that the script was completed within 2 seconds. I do another when the load is slightly higher and it is returned within 60 seconds yet the script took just 3 secs to complete from when it began.

The scenario above is easily repeatable and not due to my systems connection as I have tried on many different machines and VPSs, every time there is a huge delay before the script is even started.

Is there anything I could post that would help to identify this as 3 req/s seems very slow to me for a dedicated server.

Cheers.

 

[icaravans]

I forgot to mention that the long added delay to the returned data is not seen on the different domains on the server just the one with the high load.

I thought that a load sufficient to slow down one domain would slow down the whole server not just a single domain.

Also the figure for "requests currently being processed" I feel is high more because of a backlog than the load being overly high right now.

EDIT: I also forgot to mention that even static content of a very simple html page will timeout when the domain is bogged down but obviously not on the other domains.

 

[Oliver Muir]

Did you ever manage to find out what was causing this?

Have a very similar setup and configuration to you and was experiencing similar problems with concurrent users affecting just one domain but all others were fine.

I've set httpd maxClients to 512, fcgi maxProcesses to 500 & increased mysql max_connections to 250. So far in my load testing everything seems to be working fine for me now.