CentOS 5 64bit Plesk 12.5.30 #MU 15: nginx segfaults

[burnley]

Hi,

From Thursday I'm seeing errors like this in/var/log/nginx/error_log and nginx started to return no data to the clients:
[...]
2015/12/21 09:12:36 [alert] 2035#0: worker process 18278 exited on signal 11
2015/12/21 09:12:36 [alert] 2035#0: worker process 18279 exited on signal 11
2015/12/21 09:12:36 [alert] 2035#0: worker process 18282 exited on signal 11
2015/12/21 09:12:37 [alert] 2035#0: worker process 18284 exited on signal 11
I've enabled core dumps and the backtrace shows (no debugging symbols unfortunately):
gdb /usr/sbin/nginx /var/tmp/core.9342
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/nginx...(no debugging symbols found)...done.
[New Thread 9342]
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff1fbc3000
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000042eb0e in fflush ()
(gdb) bt
#0 0x000000000042eb0e in fflush ()
#1 0x000000000048a24d in ?? ()
#2 0x000000000049f3ea in ssl3_accept ()
#3 0x000000000042d6cf in fflush ()
#4 0x000000000042d882 in fflush ()
#5 0x000000000042a5ce in fflush ()
#6 0x0000000000422bbe in fflush ()
#7 0x000000000042862b in fflush ()
#8 0x0000000000426e10 in fflush ()
#9 0x000000000042939c in fflush ()
#10 0x000000000040a60a in fflush ()
#11 0x00007f68a39419f4 in __libc_start_main () from /lib64/libc.so.6
#12 0x0000000000408c69 in fflush ()
#13 0x00007fff1fa412f8 in ?? ()
#14 0x0000000000000000 in ?? ()

Looking it up I found there were some crashes related to OpenSSL, which should now be fixed. Other information that could help:
- This is the only CentOS 5 in our hosting environment with this issue, all the other ones are fine.
- System information:
1. uname -a
Linux cheaper01.quick.net.au 2.6.32-042stab092.2 #1 SMP Tue Jul 8 10:35:55 MSK 2014 x86_64 x86_64 x86_64 GNU/Linux
2.
cat /usr/local/psa/version
12.5.30 CentOS 5 1205151127.17
3.
rpm -q sw-nginx openssl
sw-nginx-1.9.4-centos5.15091113
openssl-0.9.8e-37.el5_11
4. I tried several configuration changes:
- reconfiguring the nginx templates with ssl_session_cache shared:SSL:10m, previously it was ssl_session_cache shared:SSL:50m
- rebuilt the server and all the domains configuration with:
/usr/local/psa/admin/sbin/httpdmng --reconfigure-server
/usr/local/psa/admin/sbin/httpdmng --reconfigure-all

After the changes above I restarted nginx and I'm waiting for it to crash again.
Is there any way I get get a sw-nginx Plesk package with debugging enabled? Or, at least the debuginfo package?

 

[burnley]

nginx -V
nginx version: nginx/1.9.4
built with OpenSSL 1.0.1p 9 Jul 2015
TLS SNI support enabled
configure arguments: --prefix=/usr/share --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --user=nginx --group=nginx --with-ipv6 --with-file-aio --with-http_ssl_module --with-http_realip_module --with-http_sub_module --with-http_dav_module --with-http_gzip_static_module --with-http_stub_status_module --with-openssl=/home/builder/buildbot/nginx-1.9.4-bcos5x64/build/nginx/work/openssl-1.0.1p --with-openssl-opt='enable-tlsext zlib no-idea no-mdc2 no-rc5 no-ec no-ecdh no-ecdsa no-shared -fpic'

 

[burnley]

Update after almost 24 hours since last nginx restart:

- reconfiguring the nginx templates with ssl_session_cache shared:SSL:10m, previously it was ssl_session_cache shared:SSL:50m
- rebuilt the server and all the domains configuration with:
/usr/local/psa/admin/sbin/httpdmng --reconfigure-server
/usr/local/psa/admin/sbin/httpdmng --reconfigure-all

Following the above reconfiguration I haven't seen any more crashes.