Resolved Very slow sites on plesk 12 , nginx having 7000+ connections from server alone

[apache123456]

I'm using plesk 12 on dedicated i5 2300 16 gb ram server.

Apache is main web server and nginx is set as reverse proxy. If i stop nginx site runs ok but after a while they take too long to open but connections goes down from 7000+ to few 100.

If i start nginx site runs ok but after a while they take too long to open and connections goes up from 100 to 7000+.

 

[Bitpalast]

Any entries in your error_log for that domain?

 

[apache123456]

nginx log
[alert] 421#0: *1170005 socket() failed (24: Too many open files) while connecting to upstream
2016/10/16 14:46:29 [crit] 419#0: accept4() failed (24: Too many open files)

apache log
(104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

 

[Bitpalast]

The root cause of the issue is likely a looping asynchronous JavaScript call from your website or worse a script including itself while making a web server call. This cannot be resolved by changes on the server, but is a website issue. A script is requesting a resource over and over again, causing an infinite number of web server requests. This is the reason why Nginx and/or Apache slow down, then finally halt. This is not a Plesk issue.

The high number of requests leads to too many open file handles. You can cure the symptom by allowing more open file handles on your host machine and on the Nginx service, but curing the symptom will not heal the root cause. It will even endanger your server, because with more open file handles, the service still fails, but fails later, leading to a much higher load in the extended time in between and probably bringing down other services on the system.

So first identify the script and error that is causing the excessive high number of requests to the web server. See your access_log or that. It should display a high number of requests to a specific resource (a script, an image). Stop or repair the script that is sending these requests.

Then work through https://kb.plesk.com/de/115139 to increase the number of open file handles that your host system allows. Again, this is not a fix for the issue, it is only a recommended practice to avoid the "too many open files" issue in other cases for the future. Do not increase the allowed number of open file handles before you have fixed the issue with the script that is sending the requests.

 

[apache123456]

there are three sites and two runs on chevereto and the chevereto owner last night tried to fix the issue and he said there is no issue in the chevereto script but its the server issue.

I've increased the limits n worker connections in nginx from start but its not helping.

He ran this command netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

and the server own IP was having 7000+ connections

 

[Bitpalast]

I've increased the limits n worker connections in nginx from start but its not helping.

That is what I said. No matter how high you set the number, sooner or later it will be reached by the infinite number of requests sent to the server.

Have you checked your access_log to see what file is sending the requests and what resource is requested with it?

 

[apache123456]

Have you checked your access_log to see what file is sending the requests and what resource is requested with it?

Sorry for asking dumb question but how do i check which file is sending the request, I do have log opened.

 

[Bitpalast]

The access_log lists all files that are accessed and normally lists a "referer" that has linked to that file. If a referer is missing, it is still helpful to find out what resource is being requested, because with that resource name you can search the source code of your project to find the lines where that resource is requested.

 

[apache123456]

::1 - - [16/Oct/2016:03:25:17 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 (internal dummy connection)"
::1 - - [16/Oct/2016:03:25:17 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 (internal dummy connection)"
::1 - - [16/Oct/2016:03:25:17 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 (internal dummy connection)"
::1 - - [16/Oct/2016:03:25:17 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 (internal dummy connection)"
::1 - - [16/Oct/2016:03:25:17 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 (internal dummy connection)"

This is showing in the log for too many times.

 

[Bitpalast]

Right place, but wrong information. "Internal dummy connections" are expected behavior in your case, because your system is opening many connections. Explanation see here: http://wiki.apache.org/httpd/InternalDummyConnection

You are looking for a file, for instance something like "index.php", "contact.html", "image.jpg", "picture.gif". I recommend to ask your web developer, because individual support on this level is not appropriate in this forum. Just follow the description above to understand the issue and you'll be able to solve it. Do not trust the person who said that their script is o.k. Webservers do not request files from themselves. It must be a script on a website that requests files. If it is not on your website, it is on someone elses website. But the cause is the the same: A script is requesting resources from your server.

 

[apache123456]

5.9.111.70 - - [16/Oct/2016:09:20:19 +0200] "GET /image/a62 HTTP/1.1" 404 207 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)"

Can it be this one ?

from this ip there are way over access to server

 

[Bitpalast]

Yes, that can be a major cause.

Add these lines before other rewrite rules to your .htaccess file to block Majestic and other bad bots:
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|spot|DigExt|Sogou) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (MegaIndex.ru|majestic12|80legs|SISTRIX|HTTrack|Semrush) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (MJ12|MJ12bot|MJ12Bot|Ezooms|CCBot|TalkTalk|Ahrefs) [NC]
RewriteRule .* - [F]

It may also be a good idea to activate the fail2ban badbot jail.

 

[apache123456]

Yes, that can be a major cause.

Add these lines before other rewrite rules to your .htaccess file to block Majestic and other bad bots:
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|spot|DigExt|Sogou) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (MegaIndex.ru|majestic12|80legs|SISTRIX|HTTrack|Semrush) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (MJ12|MJ12bot|MJ12Bot|Ezooms|CCBot|TalkTalk|Ahrefs) [NC]
RewriteRule .* - [F]

It may also be a good idea to activate the fail2ban badbot jail.

I've added those rules and enabled apache bad bots but still no luck

by running this command netstat -ntu
I getting these connection which are the causing issues
tcp6 0 0 IP:7080 IP:55344 TIME_WAIT
tcp6 0 0 IP:7081 IP:35805 TIME_WAIT
tcp6 0 0 IP:7080 IP:57305 TIME_WAIT
tcp6 0 0 IP:7081 IP:35988 TIME_WAIT
tcp6 0 0 IP:7081 IP:36457 TIME_WAIT
tcp6 0 0 IP:7081 IP:34961 TIME_WAIT
tcp6 0 0 IP:7080 IP:55984 TIME_WAIT
tcp6 0 0 IP:7080 IP:56039 TIME_WAIT
tcp6 0 0 IP:7081 IP:35838 TIME_WAIT
tcp6 0 0 IP:7080 IP:56600 TIME_WAIT

 

[Bitpalast]

It's always a good idea to keep bad bots out. But if that was not causing the problem, go back to my initial guess: It's an asynchronous script call from your own website.

 

[apache123456]

Can this be of any help ?

 

[Bitpalast]

No, this only shows too many, too intensively used MySQL processes.

 

[apache123456]

Ok then How do i find that script causing the issue ?

 

[apache123456]

Warning 175.136.156.202 mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper

2016-10-16 16:54:02 Warning 188.166.216.89 mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper Apache error
2016-10-16 16:54:03 Warning 222.127.108.7 (32)Broken pipe: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function,

 

[IgorG]

Warning 175.136.156.202 mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper

2016-10-16 16:54:02 Warning 188.166.216.89 mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper Apache error
2016-10-16 16:54:03 Warning 222.127.108.7 (32)Broken pipe: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function,

Check these articles regarding this issue:

https://kb.plesk.com/en/128129
https://kb.plesk.com/en/127268

 

[apache123456]

The issue is resolved, it was bot the script issue at all. I took some time searched for script requests but there was not a single one. Other script is fine and there is no issue in chevreto script also. Both scripts are perfectly OK. Issue was with my poor server CPU I.e. i5. I ordered and setup Plesk in Xeon server with 8 threads and all is ok now. So definitely it was not by Chevreto issue at all, the chev guru is really great supporting his clients I must admit.