Error: 502 Bad Gateway nginx (104: Connection reset by peer)

[Kingsley]

healthable.org

2016-03-09 22:33:04    Error    5.104.241.192    AH01070: Error parsing script headers    Apache error
2016-03-09 22:33:23    Error    197.242.101.41    AH01070: Error parsing script headers    Apache error
2016-03-09 23:04:43    Error    197.242.101.41    AH01070: Error parsing script headers    Apache error
2016-03-09 23:05:29    Error    81.152.59.140    AH01070: Error parsing script headers, referer: http://www.healthable.org/top-4-reasons-dental-crowns-keep-breaking/    Apache error
2016-03-09 23:05:29    Error    81.152.59.140    AH01068: Got bogus version 1, referer: http://www.healthable.org/top-4-reasons-dental-crowns-keep-breaking/    Apache error
2016-03-09 23:05:29    Error    81.152.59.140    (22)Invalid argument: AH01075: Error dispatching request to :, referer: http://www.healthable.org/top-4-reasons-dental-crowns-keep-breaking/    Apache error
2016-03-09 23:14:26    Error    209.59.141.140    AH01070: Error parsing script headers    Apache error
2016-03-09 23:15:37    Error    41.189.160.47    AH01070: Error parsing script headers, referer: http://www.healthable.org/happy-womens-day-be-proud-as-a-woman/    Apache error
2016-03-09 23:15:56    Error    41.189.160.47    AH01070: Error parsing script headers, referer: http://www.healthable.org/happy-womens-day-be-proud-as-a-woman/    Apache error
2016-03-09 23:24:00    Error    46.165.195.139    4941#0: *46552 upstream timed out (110: Connection timed out) while reading response header from upstream    nginx error
2016-03-09 23:24:01    Error    67.228.213.178    4941#0: *46554 upstream timed out (110: Connection timed out) while reading response header from upstream    nginx error
2016-03-09 23:25:52    Error    66.249.65.20    4941#0: *46676 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:27:42    Error    115.178.254.84    4941#0: *46798 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:27:52    Error    70.167.36.130    4941#0: *46806 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:29:29    Error    115.178.254.84    4941#0: *46798 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:30:49    Error    41.189.161.42    4941#0: *46999 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:30:53    Error    121.97.6.11    4941#0: *47007 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:32:21    Error    50.87.144.96    4941#0: *47095 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:32:40    Error    121.97.6.11    4941#0: *47007 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:32:41    Error    121.97.6.11    4941#0: *47110 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:32:53    Error    66.249.65.20    4941#0: *47120 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:34:28    Error    121.97.6.11    4941#0: *47110 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:34:50    Error    66.249.65.20    4941#0: *47120 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:35:55    Error    202.45.119.39    4941#0: *47316 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:36:36    Error    66.249.65.20    4941#0: *47120 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:37:42    Error    202.45.119.39    4941#0: *47316 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:38:23    Error    66.249.90.9    4941#0: *47512 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:40:07    Error    41.189.161.33    4943#0: *47636 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:41:57    Error    66.249.90.9    4943#0: *47753 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:42:40    Error    66.87.98.248    4943#0: *47807 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:43:10    Error    73.244.32.44    4943#0: *47843 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:43:51    Error    66.249.65.20    4943#0: *47876 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:45:08    Error    37.105.214.91    4943#0: *47963 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:45:49    Error    66.249.65.20    4943#0: *47876 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:47:50    Error    66.249.65.20    4943#0: *47876 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:52:19    Error    108.172.93.181    4943#0: *48396 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:52:29    Error    81.85.71.60    4943#0: *48406 recv() failed (104: Connection reset by peer) while reading response header from upstream

 

[Kingsley]

mobileif.com

2016-03-09 23:17:02    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:05    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:09    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:12    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:16    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/bcgsw    Apache error
2016-03-09 23:17:19    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:23    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:29    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:35    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:39    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/bcgsw    Apache error
2016-03-09 23:17:42    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:46    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:50    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:54    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:17:58    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/bcgsw    Apache error
2016-03-09 23:18:01    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:04    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:08    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:11    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:17    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/bcgsw    Apache error
2016-03-09 23:18:20    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:24    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:28    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:31    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/    Apache error
2016-03-09 23:18:35    Error    144.76.61.21    AH01797: client denied by server configuration: /var/www/vhosts/mobileif.com/httpdocs/bcgsw    Apache error
2016-03-09 23:28:06    Error    66.249.73.224    4941#0: *46829 recv() failed (104: Connection reset by peer) while reading response header from upstream    nginx error
2016-03-09 23:48:27    Error    208.87.233.201    4943#0: *48167 recv() failed (104: Connection reset by peer) while reading response header from upstream

 

[Oliver Meyer]

Seems the problems are caused by the apache side - which apache version are you using?

 

[Kingsley]

Just for clarification - you are using apache worker + php-fpm to handle php-files and nginx as reverse proxy for static files, right?

yes

 

[Kingsley]

Seems the problems are caused by the apache side - which apache version are you using?

Server version: Apache/2.4.7 (Ubuntu)
Server built: Jan 14 2016 17:45:23

 

[Oliver Meyer]

Maybe this bug is affecting you (apache <v2.4.11): https://bz.apache.org/bugzilla/show_bug.cgi?id=57198

Have you upgraded apache before the problems started? Maybe you should try a newer apache-version.

Have also a look if you have mod_deflate enabled - if yes, disable it for testing - it can also lead to such errors under rare conditions.

 

[Kingsley]

Maybe this bug is affecting you (apache <v2.4.11): https://bz.apache.org/bugzilla/show_bug.cgi?id=57198

Have you upgraded apache before the problems started? Maybe you should try a newer apache-version.

Have also a look if you have mod_deflate enabled - if yes, disable it for testing - it can also lead to such errors under rare conditions.

I installed it no upgrade done except MARIADB that was upgraded when i did apt-get update && apt-get upgrade

 

[Kingsley]

APACHE MPM mode is in EVENTS

 

[trialotto]

@Everyone,

The general root cause of the problem is related to PHP-FPM.

A general solution, that should work for almost everybody, is

a) make sure that the latest version of PHP-FPM is installed, (and)

b) run the command: service plesk-php<xx>-fpm restart (replace <xx> with 55, 56, 70 and so on)

c) run the command: service php-fpm restart (to make sure that PHP-FPM for the OS vendor package is restarted)

d) make sure that you do not use APC or APCu (read: disable that).

If the above does not work, then it is very likely that a memory overload still is present: in order to resolve that, just stop and start Apache and Nginx.

If a stop/start sequence (not equal to a restart) of Apache and Nginx does not work, just reboot the server (to clear up memory and resume processes as usual).

If all of the above does not work, then it is very likely that you have fiddled with fpm related settings: in order to resolve that, revert to default Plesk settings.

Hope the above helps!

Regards.....

 

[Kingsley]

@Everyone,

The general root cause of the problem is related to PHP-FPM.

A general solution, that should work for almost everybody, is

a) make sure that the latest version of PHP-FPM is installed, (and)

b) run the command: service plesk-php<xx>-fpm restart (replace <xx> with 55, 56, 70 and so on)

c) run the command: service php-fpm restart (to make sure that PHP-FPM for the OS vendor package is restarted)

d) make sure that you do not use APC or APCu (read: disable that).

If the above does not work, then it is very likely that a memory overload still is present: in order to resolve that, just stop and start Apache and Nginx.

If a stop/start sequence (not equal to a restart) of Apache and Nginx does not work, just reboot the server (to clear up memory and resume processes as usual).

If all of the above does not work, then it is very likely that you have fiddled with fpm related settings: in order to resolve that, revert to default Plesk settings.

Hope the above helps!

Regards.....

Hello,

Thanks for the tips, i will try it. What i don't understand is why after i restart apache + nginx iy will be up for hours then boom offline.

 

[Kingsley]

@trialotto how do i check Latest Version of php-fpm?

Also what do you mean memory overload? The server is 4GB ram i have not used up to 2gb

 

[Kingsley]

@Everyone,

The general root cause of the problem is related to PHP-FPM.

A general solution, that should work for almost everybody, is

a) make sure that the latest version of PHP-FPM is installed, (and)

b) run the command: service plesk-php<xx>-fpm restart (replace <xx> with 55, 56, 70 and so on)

c) run the command: service php-fpm restart (to make sure that PHP-FPM for the OS vendor package is restarted)

d) make sure that you do not use APC or APCu (read: disable that).

If the above does not work, then it is very likely that a memory overload still is present: in order to resolve that, just stop and start Apache and Nginx.

If a stop/start sequence (not equal to a restart) of Apache and Nginx does not work, just reboot the server (to clear up memory and resume processes as usual).

If all of the above does not work, then it is very likely that you have fiddled with fpm related settings: in order to resolve that, revert to default Plesk settings.

Hope the above helps!

Regards.....

root@XXXX:~# service php-fpm restart
php-fpm: unrecognized service

~# service plesk-php55-fpm restart
plesk-php55-fpm: unrecognized service

~# service plesk-php56-fpm restart
* Restarting PHP FastCGI Process Manager php-fpm [ OK ]

~# service plesk-php70-fpm restart
* Restarting PHP FastCGI Process Manager php-fpm [ OK ]

I dont have APC or APCu installed

 

[Kingsley]

I just received alert that the domains are up but it has refused to open over here, i have used plesk before it was not like this, also the server has been working fine for 8 days

 

[Kingsley]

It's been up since i removed a path to an old installation and also restarted php-fpm. i still see the peer and upstream errors but the domains are working very fine for now

 

[trialotto]

@Oliver Meyer and @Kingsley,

You have resource issues with Nginx and, to be honest, settings like the one below

Sorry, was the wrong config - here the right one (/etc/sw-cp-server/config)

worker_processes  2;

events {.....}

http {......}

will not help.

In fact, it is not required to have the performance of Plesk Panel, which runs on it´s own Nginx instance, boosted by using 2 worker processes: 1 worker process will suffice and will PREVENT that other cores are being assigned to Plesk, insted of using the cores for normal operations, such as the processes of Apache (and so on).

In short, setting "worker_processes 2" in /etc/sw-cp-server/config results in less performance in (amongst others) serving websites.

Just use or revert to ""worker_processes 1" in /etc/sw-cp-server/config.

Furthermore, the settings

If you are using nginx as reverse-proxy like me, you could try my config (/etc/nginx/nginx.conf):

worker_processes auto;
worker_rlimit_nofile 50000;

in /etc/nginx/nginx.conf are equally BAD.

The above setting "worker_processes auto" essentially imply that all or most resources will be assigned to Nginx, leaving little or no memory for (amongst others) Apache.

In short, this setting "worker_processes auto" will cause insufficient CPU power (and indirectly insufficient memory) for all other processes, including Apache.

The above setting "worker_rlimit_nofile 50000" makes no sense at all, for various reasons I will not even explain here.

Just remove that Nginx directive completely.

Regards........

PS @Kingsley, you have a 4GB Ram machine, which is a little bit of memory: try to create a swap of 4GB or just upgrade to a bigger "machine", with more memory.

 

[Kingsley]

@trialotto

will not help.

In fact, it is not required to have the performance of Plesk Panel, which runs on it´s own Nginx instance, boosted by using 2 worker processes: 1 worker process will suffice and will PREVENT that other cores are being assigned to Plesk, insted of using the cores for normal operations, such as the processes of Apache (and so on).

In short, setting "worker_processes 2" in /etc/sw-cp-server/config results in less performance in (amongst others) serving websites.

Just use or revert to ""worker_processes 1" in /etc/sw-cp-server/config.

I did not edit worker_processes in /etc/sw-cp-server/config i did mine at /etc/nginx i have changed it to 1 as you said. also i have removed the other nginx directive. but what do you think about the below?

open_file_cache max=50000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;

and

fastcgi_buffers 8 128k;
    fastcgi_buffer_size 256k;
    fastcgi_read_timeout 600;
    fastcgi_send_timeout 600;
    proxy_connect_timeout       600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    send_timeout                600;

oh, by the way my swap is 4GB

 

[trialotto]

@Kingsley

the settings

open_file_cache max=50000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;

can be removed.

The settings

fastcgi_buffers 8 128k;
fastcgi_buffer_size 256k;
fastcgi_read_timeout 600;
fastcgi_send_timeout 600;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

should do no harm, but I would rather ask you to uncomment them (i.e. make them inactive, without removing the lines).

That way you would (or should) have a "clean slate" for further analysis: it has been rather obvious that some of your issues were the result from bad configuration, it is now time to revert all customizations, in order to test which of the customizations were causing the issue.

Regards....

 

[Kingsley]

open_file_cache max=50000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;

Has been commented

fastcgi_buffers 8 128k;
fastcgi_buffer_size 256k;
fastcgi_read_timeout 600;
fastcgi_send_timeout 600;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

is left as it is

 

[Kingsley]

@Kingsley

the settings

open_file_cache max=50000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;

can be removed.

I have removed this now and restarted nginx, apache, psa

The settings

fastcgi_buffers 8 128k;
fastcgi_buffer_size 256k;
fastcgi_read_timeout 600;
fastcgi_send_timeout 600;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;



should do no harm, but I would rather ask you to uncomment them (i.e. make them inactive, without removing the lines).

This is left that way, so as to monitor things for a while

That way you would (or should) have a "clean slate" for further analysis: it has been rather obvious that some of your issues were the result from bad configuration, it is now time to revert all customizations, in order to test which of the customizations were causing the issue.

Regards....

Also i have a swap of 4GB no mailserver is installed because i use mainly external mail

 

[Kingsley]

Its down again