Apache goes down OR fails to start after Plesk vhost reconfigure using websrvmng

[SvetoslavM]

Hi all,

I have built a script that setups client accounts, domains, domain aliases etc.

The script also create a custom vhost file which in used to deactivate php safe mode.

The script *asks* apache to reload its config files so the newly setup domains use their new settings.

I have tried these 2 commands and the errors are still the same:
Apache goes down OR fails to start after them.

/usr/local/psa/admin/bin/websrvmng --reconfigure-vhost --vhost-name=example.com
OR
/usr/local/psa/admin/bin/websrvmng -a -v

Technical Info:
Plesk 8.6
OS: RHEL 5 32bit
PHP: 5.1.6

Apache Log
.....
[Mon Nov 16 14:21:06 2009] [warn] child process 29712 still did not exit, sending a SIGTERM
[Mon Nov 16 14:21:06 2009] [warn] child process 29968 still did not exit, sending a SIGTERM
[Mon Nov 16 14:21:06 2009] [warn] child process 29977 still did not exit, sending a SIGTERM
[Mon Nov 16 14:21:06 2009] [warn] child process 29713 still did not exit, sending a SIGTERM
[Mon Nov 16 14:21:06 2009] [warn] child process 29714 still did not exit, sending a SIGTERM
[Mon Nov 16 14:21:06 2009] [warn] child process 29715 still did not exit, sending a SIGTERM
.......

How can I make Apache reload only certain vhost's conf file ?
It's so annoying to have Apache down. It's a dedicated server and it hosts more than 20 clients.

 

[SvetoslavM]

Today I also noticed that

[Mon Nov 23 15:42:40 2009] [notice] child pid 10524 exit signal Segmentation fault (11)
[Mon Nov 23 16:01:37 2009] [notice] child pid 13255 exit signal Segmentation fault (11)
[Mon Nov 23 16:03:52 2009] [notice] child pid 16414 exit signal Segmentation fault (11)

Could it be bug in Apache ?

 

[IgorG]

There can be reason described here http://kb.odin.com/en/260 for Red Hat based systems. Check it and maybe it will help.

 

[SvetoslavM]

Thanks Igor,

I don't think the error is related to a lower number of file descriptors.
There are 47 domains and only 3-4 of them have SSL.

fs.file-max = 205718

There are cases where Apache manages to restart.

Svetoslav

 

[IgorG]

There can be some problems with PHP configuration which can lead to Apache segfaults. Try to check first that all modules are OK with 'php -v' As I remember from my experience there can be problem with Zend, ioncube_loader...

 

[SvetoslavM]

Hi Igor,

The problem is that is doesn't happen always.
It happens at a *random* time.

Here is what I have

-bash-3.2$ php -v
PHP 5.1.6 (cli) (built: Feb 26 2009 07:01:10) 
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
    with the ionCube PHP Loader v3.1.16, Copyright (c) 2002-2006, by ionCube Ltd.
-bash-3.2$

More apache logs

[Wed Nov 25 10:21:30 2009] [notice] Graceful restart requested, doing restart
[Wed Nov 25 10:21:32 2009] [notice] Digest: generating secret for digest authentication ...
[Wed Nov 25 10:21:32 2009] [notice] Digest: done
[Wed Nov 25 10:21:32 2009] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
........
[Wed Nov 25 10:21:32 2009] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations
[Wed Nov 25 10:21:32 2009] [notice] Graceful restart requested, doing restart
[Wed Nov 25 10:21:34 2009] [notice] Digest: generating secret for digest authentication ...
[Wed Nov 25 10:21:34 2009] [notice] Digest: done
[Wed Nov 25 10:21:34 2009] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
....
[Wed Nov 25 10:21:34 2009] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations

/usr/local/psa/admin/sbin/websrvmng    --reconfigure-vhost  --vhost 'example.com'

 

[SvetoslavM]

Is it possible the Apache problem to be caused by race conditions ?
I remember seeing this error: "long lost child came home!"

Ref: (Appendix: signals and race conditions)
http://httpd.apache.org/docs/2.0/stopping.html

 

[IgorG]

If I correctly understood it is not Virtuozzo VPS? In that case it can be lack or resources issue...

 

[SvetoslavM]

Hi Igor,

Nope. It's not a VPS.
It's a RHEL 5 dedicated server.
Linux web.example.com 2.6.18-164.6.1.el5 #1 SMP Tue Oct 27 11:30:06 EDT 2009 i686 i686 i386 GNU/Linux
Plesk version 8.6.0 with unlimited domain license.

Here are some *interesting* messages from the error log file

[Thu Nov 26 13:27:00 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
....
[Thu Nov 26 13:28:18 2009] [warn] child process 16709 still did not exit, sending a SIGTERM
[Thu Nov 26 13:28:18 2009] [warn] child process 16710 still did not exit, sending a SIGTERM
....
[Thu Nov 26 13:28:20 2009] [notice] caught SIGTERM, shutting down
[Thu Nov 26 13:28:21 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
....
[Thu Nov 26 13:28:21 2009] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations
[Thu Nov 26 13:28:30 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Nov 26 13:28:30 2009] [emerg] (22)Invalid argument: mod_fcgid: can't get lock, pid: 17084
[Thu Nov 26 13:28:30 2009] [error] mod_fcgid: fcgid process manager died, restarting the server
....

This time I also had this running from another machine:

[root@dev example]# ab -kc 10 -t 300 https://secure.example.com/app/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking secure.example.com (be patient)
apr_poll: The timeout specified has expired (70007)
Total of 717 requests completed

 

[SvetoslavM]

The server has 2GB of ram and 176G free space.

[root@web slavi]# free -m
             total       used       free     shared    buffers     cached
Mem:          2025       1446        579          0        376        465
-/+ buffers/cache:        604       1421
Swap:         2047          2       2044

 

[IgorG]

Did you tried to increase MaxClients setting?

 

[SvetoslavM]

Here is what I currently have

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75 
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>