Apache2 Crashes everyday at 7am

ilijamt · Oct 21, 2015

Hello

For some reason Apache2 crashes everyday around 7am

Code:

[Tue Oct 20 07:01:43.033135 2015] [core:notice] [pid 18913] AH00060: seg fault or similar nasty error detected in the parent process
[Wed Oct 21 06:57:37.975684 2015] [core:notice] [pid 31859] AH00060: seg fault or similar nasty error detected in the parent process
[Thu Oct 22 06:58:54.488246 2015] [core:notice] [pid 13528] AH00060: seg fault or similar nasty error detected in the parent process

And it's always preceded by a graceful restart

Code:

[mpm_event:notice] [pid 13528:tid 140709733955456] AH00493: SIGUSR1 received.  Doing graceful restart

This happens everyday at around the same time always.

Executing this almost always results in a crash with the message above.

Code:

for i in seq 1 10; do echo $i; sudo service apache2 reload; sleep 2; done;

Plesk version

Code:

Product version: 12.5.30 Update #7
    Update date: 2015/10/20 21:48
     Build date: 2015/08/26 19:00
     OS version: Debian 8.2
       Revision: 344620
   Architecture: 64-bit
Wrapper version: 1.2

IgorG · Oct 21, 2015

Have you checked your cronjob for this time (7am)? What is this cron task?
Do you have Plesk installed on VPS?

ilijamt · Oct 21, 2015

IgorG said:
Have you checked your cronjob for this time (7am)? What is this cron task?
Do you have Plesk installed on VPS?

No it's a dedicated machine with Intel® Core™ i7-4770 Quad-Core Haswell and 32 GB DDR3 RAM

I'm not running any cron tasks, this started happening after upgrade to Plesk 12.5 from 12.0.18

IgorG · Oct 21, 2015

It may be not yours but system crontasks. Check it.

ilijamt · Oct 21, 2015

Also like I said if I run

Code:

for i in seq 1 10; do echo $i; sudo service apache2 reload; sleep 2; done;

It will crash again.

But here is my cron list

Code:

0       1       *       *       *       /opt/backups/pflogsum_mail_daily.sh
0       1       *       *       1       /opt/psa/libexec/modules/watchdog/cp/send-report weekly
20      1       *       *       *       /opt/psa/libexec/modules/watchdog/cp/clean-events
15      1       *       *       1       /opt/psa/libexec/modules/watchdog/cp/pack-sysstats week
15      1       1       *       *       /opt/psa/libexec/modules/watchdog/cp/pack-sysstats year
0       1       *       *       *       /opt/maldet.generate.sh
0       0       *       *       *       /opt/psa/admin/bin/php -c '/opt/psa/admin/conf/php.ini' -dauto_prepend_file=sdk.php '/opt/psa/admin/plib/modules/servershield/scripts/servershield-version-check.php'
0       1       *       *       1       /opt/psa/libexec/modules/watchdog/cp/secur-check
15      1       *       *       *       /opt/psa/libexec/modules/watchdog/cp/pack-sysstats day
*       */1     *       *       *       /usr/local/sendmailanalyzer/sa_cache -a > /dev/null 2>&1
*/10    *       *       *       *       /opt/check_interval.sh
0       1       *       *       2       /opt/backup_from_ftp_store.sh
0       3       *       *       7       /opt/psa/libexec/modules/watchdog/cp/clean-reports
0       0       *       *       0       /opt/psa/admin/bin/php -c '/opt/psa/admin/conf/php.ini' -dauto_prepend_file=sdk.php '/opt/psa/admin/plib/modules/servershield/scripts/update-servershield.php'
15      1       1       *       *       /opt/psa/libexec/modules/watchdog/cp/pack-sysstats month
10      1       *       *       *       /opt/psa/libexec/modules/watchdog/cp/clean-sysstats
MAILTO=""
0       0       *       *       *       /opt/psa/admin/bin/php -dauto_prepend_file=sdk.php '/opt/psa/admin/plib/modules/magicspam/scripts/rotate_logs.php'
MAILTO=""
*       *       *       *       *       /opt/psa/admin/bin/php -dauto_prepend_file=sdk.php '/opt/psa/admin/plib/modules/sidekick/scripts/sidekick-ping.php'
MAILTO=""
0,5,10,15,20,25,30,35,40,45,50,55       *       *       *       *       /opt/psa/admin/bin/php -dauto_prepend_file=sdk.php '/opt/psa/admin/plib/modules/magicspam/scripts/rl-manager.php'
MAILTO=""
0       0       *       *       *       /opt/psa/admin/bin/php -dauto_prepend_file=sdk.php '/opt/psa/admin/plib/modules/magicspam/scripts/ms_purge_notifications.php'
MAILTO=""
43      0,6,12,18       *       *       *       /opt/psa/admin/bin/php -dauto_prepend_file=sdk.php '/opt/psa/admin/plib/modules/magicspam/scripts/magicspam-updater.php'

ilijamt · Oct 22, 2015

Apache2 Modules

Code:

Compiled in modules:
  core.c
  mod_so.c
  mod_watchdog.c
  http_core.c
  mod_log_config.c
  mod_logio.c
  mod_version.c
  mod_unixd.c

Code:

Loaded Modules:
 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 bw_module (shared)
 cgi_module (shared)
 cgid_module (shared)
 cloudflare_module (shared)
 dav_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 fastcgi_module (shared)
 fcgid_module (shared)
 filter_module (shared)
 geoip_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 mime_module (shared)
 mpm_event_module (shared)
 negotiation_module (shared)
 perl_module (shared)
 proxy_module (shared)
 proxy_fcgi_module (shared)
 python_module (shared)
 rewrite_module (shared)
 rpaf_module (shared)
 security2_module (shared)
 setenvif_module (shared)
 socache_shmcb_module (shared)
 ssl_module (shared)
 status_module (shared)
 suexec_module (shared)
 sysenv_module (shared)
 unique_id_module (shared)
 userdir_module (shared)

IgorG · Oct 22, 2015

Do you have any specific PHP settings, additions? Maybe XCache installed or something else?

ilijamt · Oct 22, 2015

IgorG said:
It may be not yours but system crontasks. Check it.

I can't find anything that crashes.
For some reason when you do graceful restart of apache2 it occasionally stops working

IgorG said:
Do you have any specific PHP settings, additions? Maybe XCache installed or something else?

Code:

php5-xcache:
  Installed: (none)
  Candidate: 3.2.0-1
  Version table:
     3.2.0-1 0
        500 http://mirror.hetzner.de/debian/packages/ jessie/main amd64 Packages
        500 http://http.debian.net/debian/ jessie/main amd64 Packages

As for specific PHP settings, I haven't touched the settings they are default as they were.
The only modifications I've done are through the PHP settings in the Panel, and usually for Memory Limits and Error Reporting (which doesn't work, no matter what I set it to, I get notices, even though PHP Info shows the correct value for error_reporting, display_errors)

ilijamt · Oct 22, 2015

As for PHP, I've installed them from the Plesk repository

Code:

PHP 5.6.14 (cgi-fcgi) (built: Oct  5 2015 13:11:33)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v5.0.18, Copyright (c) 2002-2015, by ionCube Ltd.
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies

ilijamt · Oct 22, 2015

And the native OS binary

Code:

PHP 5.6.13-0+deb8u1 (cgi-fcgi) (built: Sep  7 2015 13:39:12)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v5.0.12, Copyright (c) 2002-2015, by ionCube Ltd.
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies

ilijamt · Oct 22, 2015

/etc/logrotate.d/apache2

Code:

/var/log/apache2/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 reload > /dev/null; \
                fi;
        endscript
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

IgorG · Oct 22, 2015

I would suggest you contact Odin Support Team. Experienced supporters will investigate and fix this issue directly on your server.

ilijamt · Oct 22, 2015

So the only cron that runs around that time is cron.daily, maybe it's the log rotate script but I'm not sure, so I added a log entry in the logrotate script

Code:

$ cat /etc/logrotate.d/apache2
/var/log/apache2/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                logger -i -t "logrotate.apache2" "postrotate"
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 reload > /dev/null; \
                fi;
        endscript
        prerotate
                logger -i -t "logrotate.apache2" "prerotate"
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

Is there nothing else that you know, that can cause this?

Is there a way to regenerate all php.ini files for all the domains ?

ilijamt · Oct 22, 2015

Looks like the crash is done by logrotate

Code:

[Tue Oct 20 07:01:43.033135 2015] [core:notice] [pid 18913] AH00060: seg fault or similar nasty error detected in the parent process
[Wed Oct 21 06:57:37.975684 2015] [core:notice] [pid 31859] AH00060: seg fault or similar nasty error detected in the parent process
[Thu Oct 22 06:58:54.488246 2015] [core:notice] [pid 13528] AH00060: seg fault or similar nasty error detected in the parent process

Code:

Oct 20 07:01:44 s1 rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="19573" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Oct 21 06:57:38 s1 rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="19573" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Oct 22 06:58:55 s1 rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="19573" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

As you can see rsyslog was HUPed right after the crash.

After this you need to kill all the Apache2 processes

Code:

killall apache2

And after that you can start the service right away.

Code:

service apache2 start

And it works, but not without cleaning all the apache2 processes.

@IgorG do you know of any reason why this would happen

IgorG · Oct 22, 2015

ilijamt said:
@IgorG do you know of any reason why this would happen

I don't know. Therefore I suggested you contact Odin Support. Investigation directly on your server is required. Not sure what we can do it in scope of forum discussion.

ilijamt · Oct 23, 2015

The problem is logrotate, I've confirmed it.

One way around it is to replace the default /etc/logrotate.d/apache2, haven't had a crash since I updated the logrotate.
Looks like the problem is in the event and with some php plugins.
In my problem it was the mime module.

Code:

/var/log/apache2/*.log {
        daily
        missingok
        rotate 24
        compress
        delaycompress
        notifempty
        create 640 root adm
        copytruncate
}

Another way to fix this is to switch from event to prefork.

If you want to stay on event you will have to disable all the modules and then enable one by one, and test if they work by issuing the following in bash

Code:

for i in seq 1 20; do echo $i; sudo service apache2 reload; sleep 2; done;

It's a longer process but if you want to stay on event you don't have a choice. Because if you stay, you will start having segfaults with the apache processes

And you can recompile the module you found faulty, but it will be overwritten on next update, unless you create your own repository with your own packages and pin it to always install from there.

Either way problem solved.

1awww.com · Dec 31, 2015

Do you have a new informations about this problem, why we have the same!

omexlu12 · Dec 31, 2015

The solution is here:
http://talk.plesk.com/threads/logrotation-cause-problems.335926/#post-791550

Apache2 Crashes everyday at 7am

ilijamt

Basic Pleskian

IgorG

Plesk addicted!

ilijamt

Basic Pleskian

IgorG

Plesk addicted!

ilijamt

Basic Pleskian

ilijamt

Basic Pleskian

IgorG

Plesk addicted!

ilijamt

Basic Pleskian

ilijamt

Basic Pleskian

ilijamt

Basic Pleskian

ilijamt

Basic Pleskian

IgorG

Plesk addicted!

ilijamt

Basic Pleskian

ilijamt

Basic Pleskian

IgorG

Plesk addicted!

ilijamt

Basic Pleskian

1awww.com

New Pleskian

omexlu12

Basic Pleskian

Similar threads