• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Resolved Update #4 cause 502 Bad Gateway?

raytracy

raytracy

Basic Pleskian
OS: CentOS 7.3.1611 with SELinux and Nginx reverse proxy enabled, all packages updated to latest version.

I updated Onyx from 17.5.3 Update #3 to Update #4 last night. 6 hours after, all of my websites report 502 Bad Gateway message. In the same time, Plesk system health status reported all services were green light.

I have try the following action:
  • Restart httpd service
  • Restart nginx service
  • Reboot system
None of them solve the issues. And I found a new error in /var/log/messages which did not show before update#4 applied (I have reviewed all messages from last month):

May 2 11:41:09 psa-nx2 setroubleshoot: SELinux is preventing /usr/sbin/nginx from unlink access on the file nginx.pid. For complete SELinux messages. run sealert -l 218283f8-020e-4684-97c8-a6c37534d8dc​

May 2 11:41:09 psa-nx2 python: SELinux is preventing /usr/sbin/nginx from unlink access on the file nginx.pid.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that nginx should be allowed unlink access on the nginx.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'nginx' --raw | audit2allow -M my-nginx#012# semodule -i my-nginx.pp#012​

I try to fix it by:
  • ausearch -c 'nginx' --raw | audit2allow -M my-nginx
system reported this error:

[Errno 2] No such file or directory: '/etc/selinux/targeted/contexts/files/file_contexts.local'​

I also found the following error in /var/log/nginx/error.log:

2017/05/02 10:03:26 [error] 23266#0: *165 connect() failed (111: Connection refused) while connecting to upstream, client: 94.23.12.220, server: , request: "GET /CFIDE/administrator/ HTTP/1.1", upstream: "http://w.x.y.z:7080/CFIDE/administrator/", host: "w.x.y.z"
2017/05/02 10:14:10 [error] 23266#0: *171 connect() failed (111: Connection refused) while connecting to upstream, client: 202.39.224.232, server: , request: "GET /admin HTTP/1.1", upstream: "http://w.x.y.z:7080/admin", host: "migotest.domain.tld", referrer: "Google"
2017/05/02 11:32:00 [alert] 23265#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:41:05 [alert] 4260#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:43:10 [alert] 11374#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:45:14 [alert] 11494#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:47:18 [alert] 11609#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)
2017/05/02 11:48:09 [alert] 13581#0: unlink() "/var/run/nginx.pid" failed (13: Permission denied)​

The unlink() failed error begin to show from 2017/05/02 05:10:22, and my Update#4 were applied at 2017/05/01 21:46.

I have no time to try more troubleshooting options since my clients waiting for sites back to online.
So I disabled Nginx reverse proxy. All of website worked fine and error 502 gone.

I would like to enable Nginx again, but don't know how to fix the issue in above, any advice?

--- here is formatted SELinux alert by sealert -l command as following:

SELinux is preventing /usr/sbin/nginx from unlink access on the file nginx.pid.

***** Plugin catchall (100. confidence) suggests **************************

If you believe that nginx should be allowed unlink access on the nginx.pid file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'nginx' --raw | audit2allow -M my-nginx
# semodule -i my-nginx.pp


Additional Information:
Source Context system_u:system_r:httpd_t:s0
Target Context system_u: object_r:var_run_t:s0
Target Objects nginx.pid [ file ]
Source nginx
Source Path /usr/sbin/nginx
Port <Unknown>
Host fqdn.mori.cloud
Source RPM Packages sw-nginx-1.11.10-centos7.17032813.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.13.1-102.el7_3.16.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name fqdn.mori.cloud
Platform Linux fqdn.mori.cloud
3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12
15:04:24 UTC 2017 x86_64 x86_64
Alert Count 12
First Seen 2017-03-17 15:57:04 CST
Last Seen 2017-05-02 11:48:09 CST
Local ID 218283f8-020e-4684-97c8-a6c37534d8dc

Raw Audit Messages
type=AVC msg=audit(1493696889.460:268): avc: denied { unlink } for pid=13581 comm="nginx" name="nginx.pid" dev="tmpfs" ino=51243 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u: object_r:var_run_t:s0 tclass=file


type=SYSCALL msg=audit(1493696889.460:268): arch=x86_64 syscall=unlink success=no exit=EACCES a0=61e06e a1=24ade58 a2=0 a3=7ffca64507a0 items=0 ppid=1 pid=13581 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=nginx exe=/usr/sbin/nginx subj=system_u:system_r:httpd_t:s0 key=(null)

Hash: nginx,httpd_t,var_run_t,file,unlink
 
Last edited:
Hi raytracy,

raytracy said:
No such file or directory: '/etc/selinux/targeted/contexts/files/file_contexts.local'
Click to expand...
You can solve this for example with ( logged in as user "root" over SSH ):
Code: 
touch /etc/selinux/targeted/contexts/files/file_contexts.local
Afterwards, repeat your "audit2allow" command, pls.


This bug is related to the current vendor packages and not to Plesk components and extensions. Related bug - report has been created ( and confirmed ) at : => Bug 1395778 – file_contexts.local not present even though it should be
 
Thanks for your help, SELinux issue has been fixed by touch command, but Error 502 still persist if I enable Nginx as reverse proxy.

I found the nginx generated following error log:

2017/05/02 21:54:42 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.domain.tld, request: "GET / HTTP/1.1", upstream: "http://w.x.y.z:7080/", host: "psa-nx2.domain.tld"
2017/05/02 21:54:44 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.mori.cloud, request: "GET /favicon.ico HTTP/1.1", upstream: "http://w.x.y.z:7080/favicon.ico", host: "psa-nx2.domain.tld", referrer: "http://psa-nx2.domain.tld/"​

It looks like nginx has difficult to connect apache via local port 7080.
So I look at the netstat and got the following:

>netstat -nat | grep 7080
tcp6 0 0 :::7080 :::* LISTEN​

It looks like the apache did not listen on local ipv4 port 7080 but only ipv6?
I have try to connect port 7080 inside the server (by nc command), it failed by connection refused error.

BTW, If I turned Nginx off the Apache will listen on port 80 and bind to both ipv4 and ipv6 address again:

>netstat -nat | grep 80
tcp 0 0 w.x.y.z:80 61.219.246.200:53501 SYN_RECV
tcp 0 0 w.x.y.z:80 61.219.246.200:53502 SYN_RECV
tcp6 0 0 :::80 :::* LISTEN
tcp6 0 0 :::8880 :::* LISTEN​

note: I switch the nginx on/off via /usr/local/psa/admin/sbin/nginxmng command.

I have try to restart httpd and nginx service but issues still persist.
Any advice to next step for troubleshooting?
 
Last edited:
raytracy said:
2017/05/02 21:54:42 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.domain.tld, request: "GET / HTTP/1.1", upstream: "http://w.x.y.z:7080/", host: "psa-nx2.domain.tld"
2017/05/02 21:54:44 [error] 31868#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 61.219.246.200, server: psa-nx2.mori.cloud, request: "GET /favicon.ico HTTP/1.1", upstream: "http://w.x.y.z:7080/favicon.ico", host: "psa-nx2.domain.tld", referrer: "http://psa-nx2.domain.tld/"
Click to expand...

This indicates, that your corresponding PHP-FPM - service might not be active/running, or/and the apache - webserver is not active/running, or not listening to the configured ports.

Pls. note, that you are able to configure your webserver settings for each (sub)domain hosted on your server at

=> Home > Domains > (SUB)YOUR-DOMAIN.COM > Apache & nginx Settings​



In addition, pls. consider to REBUILD your webserver - configuration files, after you now fixed the NGINX issue:

Recommendation = Pls. use the Plesk REPAIR utility:
Code: 
plesk repair web -y -v

or use the "old fashioned way":
Code: 
/usr/local/psa/admin/sbin/httpdmng --reconfigure-all

Pls. check possible errors/issues/problems in your repair.log ( /var/log/plesk ).
 
Running plesk repair web -y -v got no error to be resolved:

upload_2017-5-3_1-1-3.png

PHP-FPM and Apache, Nginx looks running well:
(All website use PHP-FPM 7.1 only)
upload_2017-5-3_1-3-12.png
I look at /etc/httpd/conf/plesk.conf.d/server.conf when Nginx switched on, it configured port 7080 for all correct IP address:
upload_2017-5-3_1-7-19.png

But netstat show the port 7080 bind to ipv6 only:
upload_2017-5-3_1-8-32.png

It seems no error with apache httpd service:
upload_2017-5-3_1-9-27.png

I will try to tracing the apache startup process to find that why it won't bind to any ipv4 address?
In the mean time, any advice are welcome....
 
You must log in or register to reply here.

Similar threads

R
Plesk Random Crashes every few days 1-3 days
Replies
19
Views
2K
Dave W
Dave W
A
Issue 502 bad gateway error after migrating to alma linux and updating plesk
Replies
8
Views
2K
Bitpalast
Bitpalast
Lrnt
Resolved Nginx error each hour (502) peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream
Replies
3
Views
13K
weltonw
W
F
Issue SELinux error message - write access on the file passwd.db
Replies
0
Views
819
Freddie Bleiweiss
F
danami
Anacron job 'cron.daily' /etc/cron.daily/logrotate errors
Replies
2
Views
2K
danami
danami
Back
Top