502 bad gateway using Nginx

[ASTOKES]

Hi, I wonder if anyone can help me understand this...

Today at 12.50 pm my (ubuntu 14.04.1 LTS) server running plesk 12.0.18 suddenly stopped serving websites. Any attempts to access a webpage on any vhost returned the
502 bad gateway using Nginx
error messsage.
When I looked at the nginx logs, I found messages like:
2014/10/30 14:48:00 [error] 3099#0: *101 connect() failed (111: Connection refused) while connecting to upstream, client: m.m.m.m, server: , request: "GET /server-status HTTP/1.0", upstream: "http://n.n.n.n:7080/server-status", host: "n.n.n.n"

Restarting apache & nginx (/etc/init.d/apache2 restart & /etc/init.d/nginx restart) made no difference and I had to reboot to restore http access.
Strangely if I browsed with port 7080 specified (i.e. http://n.n.n.n:7080), this worked fine for all vhosts, so apache (which I believe is listening on this port while nginx listens on port 80) is working fine - nginx is just not able to pass stuff to it?

It would be good to be ready for this if it happens again, so if anyone can shed any light on the subject, I would be grateful...

Thanks for reading
theWoosh

 

[IgorG]

Try to increase the FastCGI buffer size to resolve the issue as below:

# cat /etc/sw-cp-server/config|grep 'fastcgi_buffer'
fastcgi_buffers 128 2048k;
fastcgi_buffer_size 2048k;

 

[ASTOKES]

Thanks I added this to /etc/sw-cp-server/config where the current settings were and restarted nginx - any other services I need to restart?
Hopefully the problem will not recur.

 

[ASTOKES]

This didn't work - nginx is again reporting 502 error on all vhosts.... what next?

 

[ASTOKES]

...at the moment the only thing I can do to get sites up again is reboot the server - restarting nginx and apache has no effect...
Pretty stuck - I could use some help...

 

[ASTOKES]

...after making the changes on Friday to /etc/sw-cp-server/config I only restarted nginx & apache. Looking elsewhere (http://kb.odin.com/en/118757) the suggestion is to perform:
# /etc/init.d/sw-cp-server restart
# /etc/init.d/sw-engine restart
...after making the changes, which I hadn't done. Is it possible that the changes I made had therefore not taken effect? ...and the subsequent reboot will have? In which case the problem may actually be solved?...

 

[ASTOKES]

...noticed I also got a stream of :
[emerg] 29991#0: open() "/var/www/vhosts/system/.........log" failed (13: Permission denied)
messages in the nginx log for every domain and every access and error log before the
connect() failed (111: Connection refused) while connecting to upstream
...messages started to appear. Have absolutely no idea what this means and information on this problem is scarce....

 

[ASTOKES]

... these proxy logs are all owned by nginx user, so no idea why this is occurring...

 

[ASTOKES]

clarification: I realise that the problem referred to in http://kb.odin.com/en/118757 mentioned above is relating to when accessing the control panel produces this error - this is not the case here - the control panel continues to work fine, it is all over vhosts sites that don't (so presumably the sw-cp-server and sw-engine restarts are irrelevant)...

 

[ASTOKES]

OK I have found the source of the problem.
at 5.30 this morning I see that fail2ban has added a jail 'plesk-apache-badbot' and banned access from itself!
I have added the localhost IP address to the fail2ban whitelist and will watch to see if this fixes the problem.
Thanks to Thomas Becker http://talk.plesk.com/threads/nginx-502-bad-gateway.312640/ for flagging the problem...

 

[UFHH01]

If you experience problems with permissions on your logfiles, please be sure, that in the standard "/etc/nginx/nginx.conf" the setting "user nginx;" is uncommented to "#user nginx;" and check that nginx is in the group "www-data" as well ( command: grep nginx /etc/group ). You might use FastCGI on your server as well and then should add "psacln" to the group nginx to avoid failures.

usermod -a -G nginx www-data
and
usermod -a -G psacln nginx

 

[ASTOKES]

Hi Thanks for your input - when you say 'uncommented to "#user nginx"' that seems a bit confusing. Do you mean remove the hash to uncomment this line? If so I will as it is currently commented...

 

[UFHH01]

The initial setting is: user nginx;

... but this can be changed to: #user nginx;

"Uncomment" always means, that you do not want a command line to be valid. In such cases you put a "#" in front of this line, rather then to delete the command line.

 

[ASTOKES]

my config is already #user nginx so no change needed...

 

[ASTOKES]

Just FYI google says:

'uncomment
verb
COMPUTING
past tense: uncommented; past participle: uncommented

1. change (a piece of text within a program) from being a comment to be part of the program that is run by the computer'

...so the opposite meaning to yours... I always say 'comment out'

 

[ASTOKES]

...to clarify - I always say 'comment out' to mean 'make a line a commment', and 'uncomment' to mean make it live again (as in the dictionary definition above)...