Resolved Update #4 cause 502 Bad Gateway?

[raytracy]

OS: CentOS 7.3.1611 with SELinux and Nginx reverse proxy enabled, all packages updated to latest version.

I updated Onyx from 17.5.3 Update #3 to Update #4 last night. 6 hours after, all of my websites report 502 Bad Gateway message. In the same time, Plesk system health status reported all services were green light.

I have try the following action:

• Restart httpd service
• Restart nginx service
• Reboot system

None of them solve the issues. And I found a new error in /var/log/messages which did not show before update#4 applied (I have reviewed all messages from last month):

May 2 11:41:09 psa-nx2 setroubleshoot: SELinux is preventing /usr/sbin/nginx from unlink access on the file nginx.pid. For complete SELinux messages. run sealert -l 218283f8-020e-4684-97c8-a6c37534d8dc​

May 2 11:41:09 psa-nx2 python: SELinux is preventing /usr/sbin/nginx from unlink access on the file nginx.pid.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that nginx should be allowed unlink access on the nginx.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'nginx' --raw | audit2allow -M my-nginx#012# semodule -i my-nginx.pp#012​

I try to fix it by:

• ausearch -c 'nginx' --raw | audit2allow -M my-nginx

system reported this error:

[Errno 2] No such file or directory: '/etc/selinux/targeted/contexts/files/file_contexts.local'​

I also found the following error in /var/log/nginx/error.log:

2017/05/02 10:03:26 [error] 23266#0: *165 connect() failed (111: Connection refused) while connecting to upstream, client: 94.23.12.220, server: , request: "GET /CFIDE/administrator/ HTTP/1.1", upstream: "http://w.x.y.z:7080/CFIDE/administrator/", host: "w.x.y.z"
2017/05/02 10:14:10 [error] 23266#0: *171 connect() failed (111: Connection refused) while connecting to upstream, client: 202.39.224.232, server: , request: "GET /admin HTTP/1.1", upstream: "http://w.x.y.z:7080/admin", host: "migotest.domain.tld", referrer: "Google"
2017/05/02 11:32:00 [alert] 23265#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:41:05 [alert] 4260#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:43:10 [alert] 11374#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:45:14 [alert] 11494#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:47:18 [alert] 11609#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:48:09 [alert] 13581#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)​

The unlink() failed error begin to show from 2017/05/02 05:10:22, and my Update#4 were applied at 2017/05/01 21:46.

I have no time to try more troubleshooting options since my clients waiting for sites back to online.
So I disabled Nginx reverse proxy. All of website worked fine and error 502 gone.

I would like to enable Nginx again, but don't know how to fix the issue in above, any advice?

--- here is formatted SELinux alert by sealert -l command as following:

SELinux is preventing /usr/sbin/nginx from unlink access on the file nginx.pid.

***** Plugin catchall (100. confidence) suggests **************************

If you believe that nginx should be allowed unlink access on the nginx.pid file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'nginx' --raw | audit2allow -M my-nginx
# semodule -i my-nginx.pp


Additional Information:
Source Context system_u:system_r:httpd_t:s0
Target Context system_u: object_r:var_run_t:s0
Target Objects nginx.pid [ file ]
Source nginx
Source Path /usr/sbin/nginx
Port <Unknown>
Host fqdn.mori.cloud
Source RPM Packages sw-nginx-1.11.10-centos7.17032813.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.13.1-102.el7_3.16.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name fqdn.mori.cloud
Platform Linux fqdn.mori.cloud
3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12
15:04:24 UTC 2017 x86_64 x86_64
Alert Count 12
First Seen 2017-03-17 15:57:04 CST
Last Seen 2017-05-02 11:48:09 CST
Local ID 218283f8-020e-4684-97c8-a6c37534d8dc

Raw Audit Messages
type=AVC msg=audit(1493696889.460:268): avc: denied { unlink } for pid=13581 comm="nginx" name="nginx.pid" dev="tmpfs" ino=51243 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u: object_r:var_run_t:s0 tclass=file


type=SYSCALL msg=audit(1493696889.460:268): arch=x86_64 syscall=unlink success=no exit=EACCES a0=61e06e a1=24ade58 a2=0 a3=7ffca64507a0 items=0 ppid=1 pid=13581 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=nginx exe=/usr/sbin/nginx subj=system_u:system_r:httpd_t:s0 key=(null)

Hash: nginx,httpd_t,var_run_t,file,unlink

 

[UFHH01]

Hi raytracy,

No such file or directory: '/etc/selinux/targeted/contexts/files/file_contexts.local'

You can solve this for example with ( logged in as user "root" over SSH ):

touch /etc/selinux/targeted/contexts/files/file_contexts.local

Afterwards, repeat your "audit2allow" command, pls.


This bug is related to the current vendor packages and not to Plesk components and extensions. Related bug - report has been created ( and confirmed ) at : => Bug 1395778 – file_contexts.local not present even though it should be

 

[raytracy]

Thanks for your help, SELinux issue has been fixed by touch command, but Error 502 still persist if I enable Nginx as reverse proxy.

I found the nginx generated following error log:

2017/05/02 21:54:42 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.domain.tld, request: "GET / HTTP/1.1", upstream: "http://w.x.y.z:7080/", host: "psa-nx2.domain.tld"
2017/05/02 21:54:44 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.mori.cloud, request: "GET /favicon.ico HTTP/1.1", upstream: "http://w.x.y.z:7080/favicon.ico", host: "psa-nx2.domain.tld", referrer: "http://psa-nx2.domain.tld/"​

It looks like nginx has difficult to connect apache via local port 7080.
So I look at the netstat and got the following:

>netstat -nat | grep 7080
tcp6 0 0 :::7080 :::* LISTEN​

It looks like the apache did not listen on local ipv4 port 7080 but only ipv6?
I have try to connect port 7080 inside the server (by nc command), it failed by connection refused error.

BTW, If I turned Nginx off the Apache will listen on port 80 and bind to both ipv4 and ipv6 address again:

>netstat -nat | grep 80
tcp 0 0 w.x.y.z:80 61.219.246.200:53501 SYN_RECV
tcp 0 0 w.x.y.z:80 61.219.246.200:53502 SYN_RECV
tcp6 0 0 :::80 :::* LISTEN
tcp6 0 0 :::8880 :::* LISTEN​

note: I switch the nginx on/off via /usr/local/psa/admin/sbin/nginxmng command.

I have try to restart httpd and nginx service but issues still persist.
Any advice to next step for troubleshooting?

 

[UFHH01]

2017/05/02 21:54:42 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.domain.tld, request: "GET / HTTP/1.1", upstream: "http://w.x.y.z:7080/", host: "psa-nx2.domain.tld"
2017/05/02 21:54:44 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.mori.cloud, request: "GET /favicon.ico HTTP/1.1", upstream: "http://w.x.y.z:7080/favicon.ico", host: "psa-nx2.domain.tld", referrer: "http://psa-nx2.domain.tld/"

This indicates, that your corresponding PHP-FPM - service might not be active/running, or/and the apache - webserver is not active/running, or not listening to the configured ports.

Pls. note, that you are able to configure your webserver settings for each (sub)domain hosted on your server at

=> Home > Domains > (SUB)YOUR-DOMAIN.COM > Apache & nginx Settings​

In addition, pls. consider to REBUILD your webserver - configuration files, after you now fixed the NGINX issue:

Recommendation = Pls. use the Plesk REPAIR utility:

plesk repair web -y -v

or use the "old fashioned way":

/usr/local/psa/admin/sbin/httpdmng --reconfigure-all

Pls. check possible errors/issues/problems in your repair.log ( /var/log/plesk ).

 

[raytracy]

Running plesk repair web -y -v got no error to be resolved:



PHP-FPM and Apache, Nginx looks running well:
(All website use PHP-FPM 7.1 only)

I look at /etc/httpd/conf/plesk.conf.d/server.conf when Nginx switched on, it configured port 7080 for all correct IP address:


But netstat show the port 7080 bind to ipv6 only:


It seems no error with apache httpd service:


I will try to tracing the apache startup process to find that why it won't bind to any ipv4 address?
In the mean time, any advice are welcome....

 

[raytracy]

This issues has been solved after I applied these package update: