• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Resolved Apache reload/graceful restart causes Apache segfault

Chris1

Chris1

Regular Pleskian
Hello,

I had clients contacting me this morning advising that their websites were offline, I then checked the running services and Apache was offline, I restarted and all ok but I want to know why.

This is the Apache error log leading up to the problem.

The reason why I think that it had something to do with Plesk update 8 is because I also received an email at 05:35 advising that an update had just been performed.

Code: 
[Mon Nov 02 05:34:10.429639 2015] [mpm_event:notice] [pid 262291:tid 139778723899456] AH00492: caught SIGWINCH, shutting down gracefully
[Mon Nov 02 05:34:15.138252 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: use Min UID 10000
[Mon Nov 02 05:34:15.138386 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: use filter for LVE exit
[Mon Nov 02 05:34:15.138405 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: version 1.0-23. LVE mechanism enabled
[Mon Nov 02 05:34:15.138425 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: found apr extention version 3
[Mon Nov 02 05:34:15.138440 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Nov 02 05:34:15.148857 2015] [suexec:notice] [pid 195462:tid 140631026325568] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 02 05:34:16.358568 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH01909: RSA certificate configured for roundcube.webmail:443 does NOT include an ID which matches the server name
[Mon Nov 02 05:34:16.358996 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Mon Nov 02 05:34:16.359547 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH01909: RSA certificate configured for default-43_229_61_136:443 does NOT include an ID which matches the server name
[Mon Nov 02 05:34:16.359896 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Mon Nov 02 05:34:16.359979 2015] [:notice] [pid 195462:tid 140631026325568] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
[Mon Nov 02 05:34:16.359991 2015] [:notice] [pid 195462:tid 140631026325568] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Mon Nov 02 05:34:16.359997 2015] [:notice] [pid 195462:tid 140631026325568] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Mon Nov 02 05:34:16.360002 2015] [:notice] [pid 195462:tid 140631026325568] ModSecurity: LUA compiled version="Lua 5.1"
[Mon Nov 02 05:34:16.360005 2015] [:notice] [pid 195462:tid 140631026325568] ModSecurity: LIBXML compiled version="2.9.1"
[Mon Nov 02 05:34:16.360009 2015] [:notice] [pid 195462:tid 140631026325568] ModSecurity: Original server signature: Apache
[Mon Nov 02 05:34:16.360013 2015] [:notice] [pid 195462:tid 140631026325568] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Mon Nov 02 05:34:16.394775 2015] [so:warn] [pid 195462] AH01574: module actions_module is already loaded, skipping
[Mon Nov 02 05:34:16.397730 2015] [so:warn] [pid 195462] AH01574: module headers_module is already loaded, skipping
[Mon Nov 02 05:34:16.398073 2015] [so:warn] [pid 195462] AH01574: module logio_module is already loaded, skipping
[Mon Nov 02 05:34:16.399555 2015] [so:warn] [pid 195462] AH01574: module suexec_module is already loaded, skipping
[Mon Nov 02 05:34:16.423980 2015] [so:warn] [pid 195462:tid 140631026325568] AH01574: module fcgid_module is already loaded, skipping
[Mon Nov 02 05:34:16.615344 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: use Min UID 10000
[Mon Nov 02 05:34:16.615452 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: use filter for LVE exit
[Mon Nov 02 05:34:16.615510 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: version 1.0-23. LVE mechanism enabled
[Mon Nov 02 05:34:16.615521 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: found apr extention version 3
[Mon Nov 02 05:34:16.615526 2015] [:notice] [pid 195462:tid 140631026325568] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Nov 02 05:34:16.622176 2015] [auth_digest:notice] [pid 195462:tid 140631026325568] AH01757: generating secret for digest authentication ...
[Mon Nov 02 05:34:16.623064 2015] [lbmethod_heartbeat:notice] [pid 195462:tid 140631026325568] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 02 05:34:16.654636 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH01909: RSA certificate configured for roundcube.webmail:443 does NOT include an ID which matches the server name
[Mon Nov 02 05:34:16.654876 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Mon Nov 02 05:34:16.655134 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH01909: RSA certificate configured for default-43_229_61_136:443 does NOT include an ID which matches the server name
[Mon Nov 02 05:34:16.655315 2015] [ssl:warn] [pid 195462:tid 140631026325568] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Mon Nov 02 05:34:16.682154 2015] [mpm_event:notice] [pid 195462:tid 140631026325568] AH00489: Apache/2.4.6 () OpenSSL/1.0.1e-fips Apache mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 configured -- re
[Mon Nov 02 05:34:16.682264 2015] [core:notice] [pid 195462:tid 140631026325568] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Mon Nov 02 05:53:32.771108 2015] [mpm_event:notice] [pid 195462:tid 140631026325568] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 02 05:53:33.661634 2015] [core:notice] [pid 195462] AH00060: seg fault or similar nasty error detected in the parent process
[Mon Nov 02 05:53:44.211554 2015] [fcgid:error] [pid 195464:tid 140631026325568] FastCGI process 214468 still did not exit, terminating forcefully
[Mon Nov 02 05:53:44.215411 2015] [fcgid:error] [pid 195464:tid 140631026325568] FastCGI process 214423 still did not exit, terminating forcefully

If someone from Plesk could explain what happened here that would be fantastic.

If it helps I am running the following:

PLesk 12.5.30 (Updated to MU 8 at ~05:35 today)
CloudLinux 7.1
Using Apache with nginx reverse proxy
 
Hi Chris1,

could you please post the output of "apachectl -M" for further investigations?

Do manual gracefull restarts of apache result as well in segfaults? ( Please test the command: "apache2ctl graceful && apache2ctl graceful && apache2ctl graceful && apache2ctl graceful" and have another look at your logs, please ).

Due to the fact that your eMail is from around 05:35, including a gracefull restart at 05:34, without any segfaults and the next gracefull restart of apache was at 05:53 ( WITH the segfault! ), I doubt that the Plesk updates were causing your segfault at all. There must be another cause/reason, why apache was restartet... does your daily/weekly/monthly logrotate starts about this time on your server? If so, please consider to post as well your apache2 - logrotate file ( "/etc/logrotate.d/apache2" ). Consider as well a manual logrotate start with the command "/usr/sbin/logrotate -v -f /etc/logrotate.d/apache2 > /var/log/test_logrotate.log 2>&1".

You could as well capture your segfault, for further investigations, using "strace" for example: First, please stop apache completely and then start it with the command "strace -ff -o /var/log/test_trace.log -r apachectl start" and then try several gracefull restarts and if this doesn't result in a segfault, consider again to use the manual logrotate command above, if this was causing the segfault.
 
Hi @UFHH01

Thank you for your reply.

Here is the output on "apachectl -M":
Code: 
[Mon Nov 02 10:21:47.801153 2015] [so:warn] [pid 238769] AH01574: module actions_module is already loaded, skipping
[Mon Nov 02 10:21:47.958509 2015] [so:warn] [pid 238769] AH01574: module headers_module is already loaded, skipping
[Mon Nov 02 10:21:47.982459 2015] [so:warn] [pid 238769] AH01574: module logio_module is already loaded, skipping
[Mon Nov 02 10:21:48.044847 2015] [so:warn] [pid 238769] AH01574: module suexec_module is already loaded, skipping
[Mon Nov 02 10:21:48.442871 2015] [so:warn] [pid 238769:tid 140653948700736] AH01574: module fcgid_module is already loaded, skipping
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
suexec_module (shared)
actions_module (shared)
logio_module (shared)
headers_module (shared)
access_compat_module (shared)
alias_module (shared)
allowmethods_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_anon_module (shared)
authn_core_module (shared)
authn_dbd_module (shared)
authn_dbm_module (shared)
authn_file_module (shared)
authn_socache_module (shared)
authz_core_module (shared)
authz_dbd_module (shared)
authz_dbm_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_owner_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cache_module (shared)
cache_disk_module (shared)
data_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
dumpio_module (shared)
echo_module (shared)
env_module (shared)
expires_module (shared)
ext_filter_module (shared)
filter_module (shared)
include_module (shared)
info_module (shared)
log_config_module (shared)
mime_magic_module (shared)
mime_module (shared)
negotiation_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
setenvif_module (shared)
slotmem_plain_module (shared)
slotmem_shm_module (shared)
socache_dbm_module (shared)
socache_memcache_module (shared)
socache_shmcb_module (shared)
status_module (shared)
substitute_module (shared)
unique_id_module (shared)
unixd_module (shared)
userdir_module (shared)
version_module (shared)
vhost_alias_module (shared)
usertrack_module (shared)
speling_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
lua_module (shared)
mpm_event_module (shared)
proxy_module (shared)
lbmethod_bybusyness_module (shared)
lbmethod_byrequests_module (shared)
lbmethod_bytraffic_module (shared)
lbmethod_heartbeat_module (shared)
proxy_ajp_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_fdpass_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_scgi_module (shared)
ssl_module (shared)
systemd_module (shared)
cgid_module (shared)
perl_module (shared)
fcgid_module (shared)
aclr_module (shared)
cloudflare_module (shared)
hostinglimits_module (shared)
security2_module (shared)
sysenv_module (shared)

Here is my log after doing graceful restarts of Apache (apachectl graceful && apachectl graceful && apachectl graceful && apachectl graceful):
Code: 
[Mon Nov 02 10:24:06.813134 2015] [mpm_event:notice] [pid 233851:tid 139854254241856] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 02 10:24:06.917033 2015] [so:warn] [pid 233851] AH01574: module actions_module is already loaded, skipping
[Mon Nov 02 10:24:06.920090 2015] [so:warn] [pid 233851] AH01574: module headers_module is already loaded, skipping
[Mon Nov 02 10:24:06.920450 2015] [so:warn] [pid 233851] AH01574: module logio_module is already loaded, skipping
[Mon Nov 02 10:24:06.921800 2015] [so:warn] [pid 233851] AH01574: module suexec_module is already loaded, skipping
[Mon Nov 02 10:24:06.944962 2015] [so:warn] [pid 233851:tid 139854254241856] AH01574: module fcgid_module is already loaded, skipping
[Mon Nov 02 10:24:07.151805 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: use Min UID 10000
[Mon Nov 02 10:24:07.151897 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: use filter for LVE exit
[Mon Nov 02 10:24:07.151906 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: version 1.0-23. LVE mechanism enabled
[Mon Nov 02 10:24:07.151920 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: found apr extention version 3
[Mon Nov 02 10:24:07.151942 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Nov 02 10:24:07.160532 2015] [auth_digest:notice] [pid 233851:tid 139854254241856] AH01757: generating secret for digest authentication ...
[Mon Nov 02 10:24:07.161579 2015] [lbmethod_heartbeat:notice] [pid 233851:tid 139854254241856] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 02 10:24:07.197679 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for roundcube.webmail:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:24:07.197925 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:24:07.198190 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for default-43_229_61_136:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:24:07.198370 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Mon Nov 02 10:24:07.208782 2015] [mpm_event:notice] [pid 233851:tid 139854254241856] AH00489: Apache/2.4.6 () OpenSSL/1.0.1e-fips Apache mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 configured -- resuming normal operations
[Mon Nov 02 10:24:07.208847 2015] [core:notice] [pid 233851:tid 139854254241856] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Mon Nov 02 10:24:07.401795 2015] [mpm_event:notice] [pid 233851:tid 139854254241856] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 02 10:24:07.480572 2015] [so:warn] [pid 233851] AH01574: module actions_module is already loaded, skipping
[Mon Nov 02 10:24:07.483504 2015] [so:warn] [pid 233851] AH01574: module headers_module is already loaded, skipping
[Mon Nov 02 10:24:07.483828 2015] [so:warn] [pid 233851] AH01574: module logio_module is already loaded, skipping
[Mon Nov 02 10:24:07.485400 2015] [so:warn] [pid 233851] AH01574: module suexec_module is already loaded, skipping
[Mon Nov 02 10:24:07.512279 2015] [so:warn] [pid 233851:tid 139854254241856] AH01574: module fcgid_module is already loaded, skipping
[Mon Nov 02 10:24:07.702892 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: use Min UID 10000
[Mon Nov 02 10:24:07.703051 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: use filter for LVE exit
[Mon Nov 02 10:24:07.703061 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: version 1.0-23. LVE mechanism enabled
[Mon Nov 02 10:24:07.703070 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: found apr extention version 3
[Mon Nov 02 10:24:07.703079 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Nov 02 10:24:07.710157 2015] [auth_digest:notice] [pid 233851:tid 139854254241856] AH01757: generating secret for digest authentication ...
[Mon Nov 02 10:24:07.710890 2015] [lbmethod_heartbeat:notice] [pid 233851:tid 139854254241856] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 02 10:24:07.734552 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for roundcube.webmail:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:24:07.734816 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:24:07.735201 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for default-43_229_61_136:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:24:07.735437 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Mon Nov 02 10:24:07.747877 2015] [mpm_event:notice] [pid 233851:tid 139854254241856] AH00489: Apache/2.4.6 () OpenSSL/1.0.1e-fips Apache mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 configured -- resuming normal operations
[Mon Nov 02 10:24:07.747947 2015] [core:notice] [pid 233851:tid 139854254241856] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Apache is still running.
 
Here is my /etc/logrotate.d/httpd file:
Code: 
/var/log/httpd/*log {
  missingok
  notifempty
  sharedscripts
  delaycompress
  postrotate
  /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
  endscript
}

How can I find out exactly when my log rotation happens?

I performed a test log rotation with /usr/sbin/logrotate -v -f /etc/logrotate.d/httpd > /var/log/test_logrotate.log 2>&1 and have this in my Apache error log:

Code: 
[Mon Nov 02 10:29:03.972014 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: use Min UID 10000
[Mon Nov 02 10:29:03.972108 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: use filter for LVE exit
[Mon Nov 02 10:29:03.972117 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: version 1.0-23. LVE mechanism enabled
[Mon Nov 02 10:29:03.972131 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: found apr extention version 3
[Mon Nov 02 10:29:03.972143 2015] [:notice] [pid 233851:tid 139854254241856] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Nov 02 10:29:03.985627 2015] [auth_digest:notice] [pid 233851:tid 139854254241856] AH01757: generating secret for digest authentication ...
[Mon Nov 02 10:29:03.986877 2015] [lbmethod_heartbeat:notice] [pid 233851:tid 139854254241856] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 02 10:29:04.038777 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for roundcube.webmail:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:29:04.039111 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:29:04.039578 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH01909: RSA certificate configured for default-43_229_61_136:443 does NOT include an ID which matches the server name
[Mon Nov 02 10:29:04.039881 2015] [ssl:warn] [pid 233851:tid 139854254241856] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Mon Nov 02 10:29:04.060470 2015] [mpm_event:notice] [pid 233851:tid 139854254241856] AH00489: Apache/2.4.6 () OpenSSL/1.0.1e-fips Apache mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 configured -- resuming normal operations
[Mon Nov 02 10:29:04.060628 2015] [core:notice] [pid 233851:tid 139854254241856] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

This is the output of the log rotation task:

Code: 
reading config file /etc/logrotate.d/httpd

Handling 1 logs

rotating pattern: /var/log/httpd/*log  forced from command line (no old logs will be kept)
empty log files are not rotated, old logs are removed
considering log /var/log/httpd/access_log
  log needs rotating
considering log /var/log/httpd/audit_log
  log does not need rotating
considering log /var/log/httpd/error_log
  log needs rotating
considering log /var/log/httpd/ssl_access_log
  log does not need rotating
considering log /var/log/httpd/ssl_error_log
  log does not need rotating
considering log /var/log/httpd/ssl_request_log
  log does not need rotating
considering log /var/log/httpd/suexec_log
  log needs rotating
rotating log /var/log/httpd/access_log, log->rotateCount is 0
dateext suffix '-20151102'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/httpd/access_log.1 to /var/log/httpd/access_log.2 (rotatecount 1, logstart 1, i 1),
old log /var/log/httpd/access_log.1 does not exist
renaming /var/log/httpd/access_log.0 to /var/log/httpd/access_log.1 (rotatecount 1, logstart 1, i 0),
old log /var/log/httpd/access_log.0 does not exist
log /var/log/httpd/access_log.2 doesn't exist -- won't try to dispose of it
rotating log /var/log/httpd/error_log, log->rotateCount is 0
dateext suffix '-20151102'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/httpd/error_log.1 to /var/log/httpd/error_log.2 (rotatecount 1, logstart 1, i 1),
old log /var/log/httpd/error_log.1 does not exist
renaming /var/log/httpd/error_log.0 to /var/log/httpd/error_log.1 (rotatecount 1, logstart 1, i 0),
old log /var/log/httpd/error_log.0 does not exist
log /var/log/httpd/error_log.2 doesn't exist -- won't try to dispose of it
rotating log /var/log/httpd/suexec_log, log->rotateCount is 0
dateext suffix '-20151102'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/httpd/suexec_log.1 to /var/log/httpd/suexec_log.2 (rotatecount 1, logstart 1, i 1),
old log /var/log/httpd/suexec_log.1 does not exist
renaming /var/log/httpd/suexec_log.0 to /var/log/httpd/suexec_log.1 (rotatecount 1, logstart 1, i 0),
old log /var/log/httpd/suexec_log.0 does not exist
log /var/log/httpd/suexec_log.2 doesn't exist -- won't try to dispose of it
renaming /var/log/httpd/access_log to /var/log/httpd/access_log.1
disposeName will be /var/log/httpd/access_log.1
renaming /var/log/httpd/error_log to /var/log/httpd/error_log.1
disposeName will be /var/log/httpd/error_log.1
renaming /var/log/httpd/suexec_log to /var/log/httpd/suexec_log.1
disposeName will be /var/log/httpd/suexec_log.1
running postrotate script
removing old log /var/log/httpd/access_log.1
removing old log /var/log/httpd/error_log.1
removing old log /var/log/httpd/suexec_log.1

So far I haven't been able to replicate the fault.

What does strace do? I might need to schedule a maintenance window for the last test as it will mean switching off Apache completely.
 
Last edited:
Hi Chris1,

you don't need to schedule that at all. The strace command just "traces system calls and signals"... so nothing to be worried about - your apache will run as always, but "strace" will log all, what you normally don't see. :) ( http://linux.die.net/man/1/strace ).

Actually, there is so far nothing wrong or suspicious. Please provide more informations about your current apache - version.
 
Hi @UFHH01

No worries, I'll take a look. But I have stop Apache first right?

My version of Apache is 2.4.6-31.el7_1.1.cloudlinux.
 
Hi @UFHH01

I was able to reproduce the problem by running this again:

# apachectl graceful && apachectl graceful && apachectl graceful && apachectl graceful

It stopped Apache completely.

/var/log/httpd/error_log
Code: 
[Mon Nov 02 11:12:06.686639 2015] [mpm_event:notice] [pid 233851:tid 139854254241856] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 02 11:12:06.985479 2015] [core:notice] [pid 233851] AH00060: seg fault or similar nasty error detected in the parent process
[Mon Nov 02 11:12:17.822691 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: use Min UID 10000
[Mon Nov 02 11:12:17.822907 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: use filter for LVE exit
[Mon Nov 02 11:12:17.822923 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: version 1.0-23. LVE mechanism enabled
[Mon Nov 02 11:12:17.822944 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: found apr extention version 3
[Mon Nov 02 11:12:17.822961 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Nov 02 11:12:17.838008 2015] [suexec:notice] [pid 241664:tid 140342467258432] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 02 11:12:17.909943 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH01909: RSA certificate configured for roundcube.webmail:443 does NOT include an ID which matches the server name
[Mon Nov 02 11:12:17.910191 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Mon Nov 02 11:12:17.910477 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH01909: RSA certificate configured for default-43_229_61_136:443 does NOT include an ID which matches the server name
[Mon Nov 02 11:12:17.910731 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Mon Nov 02 11:12:17.910828 2015] [:notice] [pid 241664:tid 140342467258432] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
[Mon Nov 02 11:12:17.910847 2015] [:notice] [pid 241664:tid 140342467258432] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Mon Nov 02 11:12:17.910897 2015] [:notice] [pid 241664:tid 140342467258432] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Mon Nov 02 11:12:17.910904 2015] [:notice] [pid 241664:tid 140342467258432] ModSecurity: LUA compiled version="Lua 5.1"
[Mon Nov 02 11:12:17.910909 2015] [:notice] [pid 241664:tid 140342467258432] ModSecurity: LIBXML compiled version="2.9.1"
[Mon Nov 02 11:12:17.910913 2015] [:notice] [pid 241664:tid 140342467258432] ModSecurity: Original server signature: Apache
[Mon Nov 02 11:12:17.910916 2015] [:notice] [pid 241664:tid 140342467258432] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Mon Nov 02 11:12:17.935408 2015] [so:warn] [pid 241664] AH01574: module actions_module is already loaded, skipping
[Mon Nov 02 11:12:17.937997 2015] [so:warn] [pid 241664] AH01574: module headers_module is already loaded, skipping
[Mon Nov 02 11:12:17.938338 2015] [so:warn] [pid 241664] AH01574: module logio_module is already loaded, skipping
[Mon Nov 02 11:12:17.939668 2015] [so:warn] [pid 241664] AH01574: module suexec_module is already loaded, skipping
[Mon Nov 02 11:12:17.965050 2015] [so:warn] [pid 241664:tid 140342467258432] AH01574: module fcgid_module is already loaded, skipping
[Mon Nov 02 11:12:18.154662 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: use Min UID 10000
[Mon Nov 02 11:12:18.154761 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: use filter for LVE exit
[Mon Nov 02 11:12:18.154770 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: version 1.0-23. LVE mechanism enabled
[Mon Nov 02 11:12:18.154778 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: found apr extention version 3
[Mon Nov 02 11:12:18.154784 2015] [:notice] [pid 241664:tid 140342467258432] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Nov 02 11:12:18.161215 2015] [auth_digest:notice] [pid 241664:tid 140342467258432] AH01757: generating secret for digest authentication ...
[Mon Nov 02 11:12:18.162282 2015] [lbmethod_heartbeat:notice] [pid 241664:tid 140342467258432] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 02 11:12:18.213585 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH01909: RSA certificate configured for roundcube.webmail:443 does NOT include an ID which matches the server name
[Mon Nov 02 11:12:18.213993 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Mon Nov 02 11:12:18.214458 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH01909: RSA certificate configured for default-43_229_61_136:443 does NOT include an ID which matches the server name
[Mon Nov 02 11:12:18.214677 2015] [ssl:warn] [pid 241664:tid 140342467258432] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Mon Nov 02 11:12:18.235386 2015] [core:warn] [pid 241664:tid 140342467258432] AH00098: pid file /run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Mon Nov 02 11:12:18.243727 2015] [mpm_event:notice] [pid 241664:tid 140342467258432] AH00489: Apache/2.4.6 () OpenSSL/1.0.1e-fips Apache mod_fcgid/2.3.9 mod_perl/2.0.9-dev Perl/v5.16.3 configured -- resuming normal operations
[Mon Nov 02 11:12:18.243802 2015] [core:notice] [pid 241664:tid 140342467258432] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Why is this faulting on a graceful restart?
 
Hi Chris1,

sorry Chris1, I mostly don't guess, while investigating. It's mostly only time - consuming. Please use the strace - command as suggested above and let apache crash again. You will then have a log at: "/var/log/test_trace.log", which you should investigate and/or post in your next answer.
 
Hi @UFHH01

Here is the trace log, it took me a few graceful apache restarts to replicate the problem but it eventually did it.

I had to split the log into three posts due to forum post restrictions.

Code: 
     0.000000 execve("/usr/sbin/apachectl", ["apachectl", "start"], [/* 21 vars */]) = 0
     0.000336 brk(0)                    = 0x1ff7000
     0.000057 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f347015a000
     0.000059 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
     0.000079 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
     0.000045 fstat(3, {st_mode=S_IFREG|0644, st_size=33824, ...}) = 0
     0.000032 mmap(NULL, 33824, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3470151000
     0.000028 close(3)                  = 0
     0.000043 open("/lib64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = 3
     0.000105 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\316\0\0\0\0\0\0"..., 832) = 832
     0.000043 fstat(3, {st_mode=S_IFREG|0755, st_size=174520, ...}) = 0
     0.000031 mmap(NULL, 2268928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f346fd10000
     0.000029 mprotect(0x7f346fd35000, 2097152, PROT_NONE) = 0
     0.000029 mmap(0x7f346ff35000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7f346ff35000
     0.000043 close(3)                  = 0
     0.000035 open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
     0.000059 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
     0.000029 fstat(3, {st_mode=S_IFREG|0755, st_size=19512, ...}) = 0
     0.000029 mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f346fb0c000
     0.000031 mprotect(0x7f346fb0f000, 2093056, PROT_NONE) = 0
     0.000029 mmap(0x7f346fd0e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f346fd0e000
     0.000049 close(3)                  = 0
     0.000047 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
     0.000038 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\34\2\0\0\0\0\0"..., 832) = 832
     0.000029 fstat(3, {st_mode=S_IFREG|0755, st_size=2107760, ...}) = 0
     0.000028 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3470150000
     0.000033 mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f346f74b000
     0.000030 mprotect(0x7f346f901000, 2097152, PROT_NONE) = 0
     0.000029 mmap(0x7f346fb01000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f346fb01000
     0.020435 mmap(0x7f346fb07000, 16960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f346fb07000
     0.000129 close(3)                  = 0
     0.000119 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f347014e000
     0.000098 arch_prctl(ARCH_SET_FS, 0x7f347014e740) = 0
     0.000340 mprotect(0x7f346fb01000, 16384, PROT_READ) = 0
     0.000079 mprotect(0x7f346fd0e000, 4096, PROT_READ) = 0
     0.000144 mprotect(0x7f346ff35000, 16384, PROT_READ) = 0
     0.000089 mprotect(0x6dc000, 4096, PROT_READ) = 0
     0.000055 mprotect(0x7f347015b000, 4096, PROT_READ) = 0
     0.000033 munmap(0x7f3470151000, 33824) = 0
     0.000158 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000081 open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
     0.000085 close(3)                  = 0
     0.000099 brk(0)                    = 0x1ff7000
     0.000090 brk(0)                    = 0x1ff7000
     0.000028 brk(0x2018000)            = 0x2018000
     0.000054 brk(0)                    = 0x2018000
     0.000083 getuid()                  = 0
     0.018016 getgid()                  = 0
     0.000117 geteuid()                 = 0
     0.000060 getegid()                 = 0
     0.000094 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000171 open("/proc/meminfo", O_RDONLY|O_CLOEXEC) = 3
     0.001061 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
     0.000048 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3470159000
     0.000054 read(3, "MemTotal:        2045904 kB\nMemF"..., 1024) = 1024
     0.000280 close(3)                  = 0
     0.000067 munmap(0x7f3470159000, 4096) = 0
     0.000110 rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7f346f780650}, {SIG_DFL, [], 0}, 8) = 0
     0.000098 rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7f346f780650}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7f346f780650}, 8) = 0
     0.000067 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, {SIG_DFL, [], 0}, 8) = 0
     0.000059 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, 8) = 0
     0.000033 rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, {SIG_DFL, [], 0}, 8) = 0
     0.000052 rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, 8) = 0
     0.000075 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000068 rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f346f780650}, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, 8) = 0
     0.000102 uname({sys="Linux", node="ws1.razz.net.au", ...}) = 0
     0.000131 stat("/var/log", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
     0.000082 stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
     0.000080 getpid()                  = 247943
     0.000086 getppid()                 = 247940
     0.000115 getpgrp()                 = 247940
     0.000068 rt_sigaction(SIGCHLD, {0x441090, [], SA_RESTORER|SA_RESTART, 0x7f346f780650}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7f346f780650}, 8) = 0
     0.000102 getrlimit(RLIMIT_NPROC, {rlim_cur=15917, rlim_max=15917}) = 0
     0.000179 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000070 open("/usr/sbin/apachectl", O_RDONLY) = 3
     0.000054 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fffa611bc80) = -1 ENOTTY (Inappropriate ioctl for device)
     0.017534 lseek(3, 0, SEEK_CUR)     = 0
     0.000044 read(3, "#!/bin/sh\n#\n# Licensed to the Ap"..., 80) = 80
     0.000036 lseek(3, 0, SEEK_SET)     = 0
     0.000028 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
     0.000032 fcntl(255, F_GETFD)       = -1 EBADF (Bad file descriptor)
     0.000029 dup2(3, 255)              = 255
     0.000027 close(3)                  = 0
     0.000025 fcntl(255, F_SETFD, FD_CLOEXEC) = 0
     0.000027 fcntl(255, F_GETFL)       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
     0.000032 fstat(255, {st_mode=S_IFREG|0755, st_size=4263, ...}) = 0
     0.000043 lseek(255, 0, SEEK_CUR)   = 0
     0.000048 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000057 read(255, "#!/bin/sh\n#\n# Licensed to the Ap"..., 4263) = 4263
 
Code: 
0.000069 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000035 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000034 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.030858 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000126 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000070 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000073 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000072 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000063 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000067 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000063 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000070 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000072 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000072 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.007666 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000041 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000032 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000030 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000034 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000038 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000031 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000031 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000034 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000031 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000032 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000030 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000031 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000031 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000086 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000034 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000033 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000033 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000048 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000032 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000034 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000033 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000031 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000122 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000065 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000036 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000036 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000035 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.025868 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000059 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000094 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000046 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000086 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000063 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000070 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000066 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000136 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000101 faccessat(AT_FDCWD, "/usr/bin/links", X_OK) = -1 ENOENT (No such file or directory)
     0.000108 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000088 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000064 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000067 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000069 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000080 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000063 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000066 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000088 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000058 faccessat(AT_FDCWD, "/etc/sysconfig/httpd", R_OK) = 0
     0.000111 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000083 open("/etc/sysconfig/httpd", O_RDONLY) = 3
     0.000097 fstat(3, {st_mode=S_IFREG|0644, st_size=802, ...}) = 0
     0.000063 read(3, "#\n# This file can be used to set"..., 802) = 802
     0.000061 close(3)                  = 0
     0.000063 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000075 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000061 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000068 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000067 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000063 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000058 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000069 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000069 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000068 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000077 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000090 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000066 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000065 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000060 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000072 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000085 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000077 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000074 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000062 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000060 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000066 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000069 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000069 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000068 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000066 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000145 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000061 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000035 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000033 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000033 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000033 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000069 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000086 pipe([3, 4])              = 0
     0.000099 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
     0.000064 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000062 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
     0.000066 lseek(255, -1659, SEEK_CUR) = 2604
     0.000073 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f347014ea10) = 247946
     0.000393 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000070 rt_sigaction(SIGCHLD, {0x441090, [], SA_RESTORER|SA_RESTART, 0x7f346f780650}, {0x441090, [], SA_RESTORER|SA_RESTART, 0x7f346f780650}, 8) = 0
     0.000084 close(4)                  = 0
     0.000074 read(3, "4096\n", 128)    = 5
     0.013061 read(3, "", 128)          = 0
     0.000331 close(3)                  = 0
     0.000035 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
     0.000040 rt_sigaction(SIGINT, {0x43e500, [], SA_RESTORER, 0x7f346f780650}, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, 8) = 0
     0.000043 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 247946
     0.000090 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000030 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=247946, si_status=0, si_utime=0, si_stime=0} ---
     0.000019 wait4(-1, 0x7fffa611b050, WNOHANG, NULL) = -1 ECHILD (No child processes)
     0.000032 rt_sigreturn()            = 0
     0.000030 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, {0x43e500, [], SA_RESTORER, 0x7f346f780650}, 8) = 0
     0.000105 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000046 read(255, "# --------------------          "..., 4263) = 1659
     0.000045 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.003754 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000130 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000085 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000140 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000090 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000113 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
     0.000047 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
     0.000039 setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = 0
     0.000045 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000043 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000053 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000072 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000062 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000045 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000041 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
 
Code: 
     0.000106 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000067 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000140 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000056 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000114 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000060 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000046 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000032 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000139 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
     0.000037 rt_sigprocmask(SIG_BLOCK, [CHLD], [INT CHLD], 8) = 0
     0.000029 rt_sigprocmask(SIG_SETMASK, [INT CHLD], NULL, 8) = 0
     0.000028 lseek(255, -14, SEEK_CUR) = 4249
     0.000046 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f347014ea10) = 247947
     0.000170 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000072 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
     0.000065 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000029 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
     0.000029 rt_sigaction(SIGINT, {0x43e500, [], SA_RESTORER, 0x7f346f780650}, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, 8) = 0
     0.000037 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 247947
     1.111311 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000079 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=247947, si_status=0, si_utime=0, si_stime=0} ---
     0.000074 wait4(-1, 0x7fffa611b390, WNOHANG, NULL) = -1 ECHILD (No child processes)
     0.000062 rt_sigreturn()            = 0
     0.000076 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f346f780650}, {0x43e500, [], SA_RESTORER, 0x7f346f780650}, 8) = 0
     0.000151 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000076 read(255, "\nexit $ERROR\n\n", 4263) = 14
     0.000081 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
     0.000110 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
     0.000150 exit_group(0)             = ?
     0.000249 +++ exited with 0 +++
 
Hi Chris1,

consider to use "/bin/systemctl restart httpd.service" instead of "/bin/systemctl reload httpd.service"... untill you found the cause of your issue.

Consider as well to raise the open-file -limit for the apache system user. ( /etc/security/limits.conf ), or to setup a higher limit than the standart 1024 limit.

Example:
Code: 
...
apache        soft    nofile          64000
apache        hard    nofile          64000
...

Consider as well to un-install modules, that you don't really need and use.


For further suggestion, please wait for another forum - user to answer you, but please be informed, that this doesn't seem to be a Plesk - related issue at all, so I would change the title of the thread. :)​
 
Last edited by a moderator:
Thank you for the reply, I will try your suggestions. It is a very low traffic server though, I'd be surprised if it was related to an open file limit.

I have altered the http log rotate to use restart instead of reload for now. What is the difference between these?

Is it possible to uninstall modules from within Plesk?
 
Last edited:
I had the same problem (CentOS 7, Plesk 12.5).
It started when I installed mod_security.

As far as I understand, it occurs at logrotation, because both
/etc/logrotate.d/httpd
/etc/logrotate.d/mod_security
contain a line for httpd reload.
Apparently, they happen simultaneously and cause the 'seg fault' error of apache.

Replacing 'reload' with 'restart' solved the problem.
 
In one case, changing reload to restart only in /etc/logrotate.d/mod_security solved the issue.
In other cases, I had to do it in both files.
 
Got the same problem with Apache:

Code: 
me@server:/var/log/apache2# apachectl -M

Loaded Modules:

 core_module (static)

 so_module (static)

 watchdog_module (static)

 http_module (static)

 log_config_module (static)

 logio_module (static)

 version_module (static)

 unixd_module (static)

 access_compat_module (shared)

 aclr_module (shared)

 actions_module (shared)

 alias_module (shared)

 auth_basic_module (shared)

 auth_digest_module (shared)

 authn_core_module (shared)

 authn_file_module (shared)

 authz_core_module (shared)

 authz_host_module (shared)

 authz_user_module (shared)

 autoindex_module (shared)

 bw_module (shared)

 cache_module (shared)

 cache_disk_module (shared)

 cache_socache_module (shared)

 cgi_module (shared)

 dav_module (shared)

 dav_fs_module (shared)

 dav_lock_module (shared)

 deflate_module (shared)

 dir_module (shared)

 env_module (shared)

 expires_module (shared)

 fcgid_module (shared)

 file_cache_module (shared)

 filter_module (shared)

 headers_module (shared)

 include_module (shared)

 mime_module (shared)

 mpm_event_module (shared)

 negotiation_module (shared)

 proxy_module (shared)

 proxy_fcgi_module (shared)

 remoteip_module (shared)

 rewrite_module (shared)

 security2_module (shared)

 setenvif_module (shared)

 socache_shmcb_module (shared)

 ssl_module (shared)

 status_module (shared)

 suexec_module (shared)

 sysenv_module (shared)

 unique_id_module (shared)

 userdir_module (shared)

Changed logrotation-reload-settings and will see if the problem turns up again.
 
You must log in or register to reply here.

Similar threads

WebHostingAce
Resolved Apache fails to (re)start with Atomic ModSecurity
Replies
8
Views
2K
WebHostingAce
WebHostingAce
G
Issue AH01574 - New configuration files for the Apache web server - No matches for the wildcard '*asl*.conf
Replies
0
Views
221
gmadkins
G
H
Issue Unable to start apache
Replies
1
Views
1K
Heppi75
H
T
Apache2 ModSec Error - Apache2 fails to start and AH00111 error
Replies
1
Views
987
Peter Debik
P
A
Issue zend_mm_heap corrupted
Replies
0
Views
934
Ayki
A
Back
Top