Resolved Unexpected Server Shutdown

[mkul7ra]

Server: ‪CentOS Linux 7.6.1810, Plesk Onyx Version 17.8.11
Fail2Ban + Modserurity enabled

For some time, my server is down unexpectedly, in the log files is the following:

[Sun Feb 24 09:18:16.776372 2019] [ssl:warn] [pid 9653:tid 139961038801024] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Sun Feb 24 09:18:16.778574 2019] [:notice] [pid 9653:tid 139961038801024] mod_bw : Memory Allocated 0 bytes (each conf takes 48 bytes)
[Sun Feb 24 09:18:16.778636 2019] [:notice] [pid 9653:tid 139961038801024] mod_bw : Version 0.92 - Initialized [0 Confs]
[ N 2019-02-24 09:18:16.8043 9686/T1 age/Wat/WatchdogMain.cpp:1300 ]: Starting Passenger watchdog...
[ N 2019-02-24 09:18:16.8262 9689/T1 age/Cor/CoreMain.cpp:1310 ]: Starting Passenger core...
[ N 2019-02-24 09:18:16.8266 9689/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2019-02-24 09:18:16.8376 9689/T1 age/Cor/CoreMain.cpp:985 ]: Passenger core online, PID 9689
[Sun Feb 24 09:18:16.842824 2019] [mpm_event:notice] [pid 9653:tid 139961038801024] AH00489: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Apache mod_fcgid/2.3.9 Phusion_Passenger/5.3.5 configured -- resuming normal operations
[Sun Feb 24 09:18:16.842949 2019] [core:notice] [pid 9653:tid 139961038801024] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[ N 2019-02-24 09:18:19.0947 9689/T5 age/Cor/SecurityUpdateChecker.h:517 ]: Security update check: no update found (next check in 24 hours)
[Sun Feb 24 09:59:26.393833 2019] [access_compat:error] [pid 9707:tid 139960457815808] [client 197.214.114.90:42154] AH01797: client denied by server configuration: /var/www/vhosts/default/htdocs/
[Sun Feb 24 09:59:26.395419 2019] [access_compat:error] [pid 9707:tid 139960457815808] [client 197.214.114.90:42154] AH01797: client denied by server configuration: /usr/share/httpd/noindex/index.html
[Sun Feb 24 10:00:40.933846 2019] [mpm_event:notice] [pid 9653:tid 139961038801024] AH00492: caught SIGWINCH, shutting down gracefully

[ N 2019-02-24 10:00:44.9838 9689/T8 age/Cor/CoreMain.cpp:641 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2019-02-24 10:00:44.9839 9689/T1 age/Cor/CoreMain.cpp:1216 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2019-02-24 10:00:44.9839 9689/Ta Ser/Server.h:903 ]: [ServerThr.2] Freed 0 spare client objects
[ N 2019-02-24 10:00:44.9839 9689/Ta Ser/Server.h:559 ]: [ServerThr.2] Shutdown finished
[ N 2019-02-24 10:00:44.9840 9689/Tc Ser/Server.h:903 ]: [ServerThr.3] Freed 0 spare client objects
[ N 2019-02-24 10:00:44.9840 9689/Tc Ser/Server.h:559 ]: [ServerThr.3] Shutdown finished
[ N 2019-02-24 10:00:44.9840 9689/Te Ser/Server.h:903 ]: [ServerThr.4] Freed 0 spare client objects
[ N 2019-02-24 10:00:44.9840 9689/Te Ser/Server.h:559 ]: [ServerThr.4] Shutdown finished
[ N 2019-02-24 10:00:44.9841 9689/Tg Ser/Server.h:903 ]: [ApiServer] Freed 0 spare client objects

Are there any suggestions why this is happening?

 

[Bitpalast]

SIGWINCH is normally sent when a terminal window is resized. Apache misuses the signal on restart processes. If Apache was started in the "foreground", it is possible that it crashes when something in a terminal window changes.

Besides that, you might want to check whether Apache is using MPM Prefork and if not change it to MPM Prefork instead of Event. It might already help.

 

[mkul7ra]

Thank you for your answer. I have now switched Apache to Apache MPM Prefork. Let's see if that happens again.

But not only the Apache did crash, the whole server did. And i wasnt able to restart. I then reset to an old backup. After that, I restarted the whole server to see if it is "restart safe". Since the restart then worked well without problems. Very misterious, I hope the problem is now resolved ...

 

[mkul7ra]

Helo Peter,

i mark this thread as solved now, because the server in online, since i did your suggested changes. THANKS!

 

[Bitpalast]

Normally, when the whole system crashes, you can find something unusual in /var/log/messages. Apache can crash the whole system, but normally only if it uses up all RAM and more CPU power than the processors can deliver. This can be the case when infinite loops of caching and compression are created in .htaccess configurations (e.g. infinite rewrites, back and forth, compressing and rewriting again and again). Such issues may not occur immediately, but randomly and are very hard to diagnose, because in the crash situation, Apache does not write the last log entry to error.log or access.log files that would reveal which domain was responsible for it. If a random crash occured before, it is likely that it will occur again. However, it is very unlikely a Plesk issue.

 

[Ayki]

HI All
I have experienced a similar issue 2 weeks ago. This i snot persitant problem for me but I wanted to undesrtand it better.

Reading into the diffrences between Apache MPM Prefork and Apache MPM Event Modes. I found the the optimimum set up is the Event Mode which runs more effecinetly in high traffic webapplications. The problem with this mode appers to be an Apache bug. ( Please correct me if I am wrong)

I found the following recommendation which I will try out to see if it makes a difference:

Keeping the Apache MPM Event : ( I will increase the StartServers to 50 - It is set to 10 by dafault at he moment

StackOverflow： Scoreboard is full,not at MaxRequestWorkers

1、I had this same problem. I tried different Apache versions and MPMs. I seem to get this alot with MPM Worker. Also error does not reoccur using Apache 2.2.2，Are you using cPanel? IF so try /upcp –force and increase StartServers to a higher amount like 50 as that’s all I did to get this error away.

2、Try EnableMMAP Off in 00_default_settings.conf
(Three modes of Apache work: Prefork, Worker, Event - Develop Paper)