• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Logrotation cause Problems

O

omexlu12

Basic Pleskian
Hello there and @UFHH01 ,

As posted in this thread I open a new thread to finally be able to solve my problem:
http://talk.plesk.com/threads/nginx-502-error.335745/page-2#post-791256

My system:
Debian Jessie
Plesk 12.5

Everything on the system is in default configuration.

LIst of apache modules:
core_module (static)
so_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
logio_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgi_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
fcgid_module (shared)
filter_module (shared)
headers_module (shared)
include_module (shared)
mime_module (shared)
mpm_event_module (shared)
negotiation_module (shared)
proxy_module (shared)
proxy_fcgi_module (shared)
rewrite_module (shared)
security2_module (shared)
setenvif_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)
status_module (shared)
suexec_module (shared)
unique_id_module (shared)
userdir_module (shared)
Click to expand...

LOGS (apache):
[Sat Nov 21 09:29:36.003149 2015] [ssl:warn] [pid 17504:tid 140713314006912] AH01909: roundcube.webmail:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:36.003509 2015] [ssl:warn] [pid 17504:tid 140713314006912] AH01909: lists:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:36.003773 2015] [ssl:warn] [pid 17504:tid 140713314006912] AH01909: default-XXX:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:36.004142 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
[Sat Nov 21 09:29:36.004149 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"
[Sat Nov 21 09:29:36.004153 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: PCRE compiled version="8.35 "; loaded version="8.35 2014-04-04"
[Sat Nov 21 09:29:36.004155 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: LUA compiled version="Lua 5.1"
[Sat Nov 21 09:29:36.004157 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: LIBXML compiled version="2.9.1"
[Sat Nov 21 09:29:36.004159 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: Original server signature: Apache
[Sat Nov 21 09:29:36.004161 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Sat Nov 21 09:29:36.004475 2015] [suexec:notice] [pid 17504:tid 140713314006912] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Sat Nov 21 09:29:36.085559 2015] [auth_digest:notice] [pid 17505:tid 140713314006912] AH01757: generating secret for digest authentication ...
[Sat Nov 21 09:29:37.003544 2015] [ssl:warn] [pid 17505:tid 140713314006912] AH01909: roundcube.webmail:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:37.003852 2015] [ssl:warn] [pid 17505:tid 140713314006912] AH01909: lists:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:37.004120 2015] [ssl:warn] [pid 17505:tid 140713314006912] AH01909: default-XXX:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:37.007858 2015] [core:warn] [pid 17505:tid 140713314006912] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Sat Nov 21 09:29:37.009678 2015] [mpm_event:notice] [pid 17505:tid 140713314006912] AH00489: Apache/2.4.10 (Debian) OpenSSL/1.0.1k Apache mod_fcgid/2.3.9 configured -- resuming normal operations
[Sat Nov 21 09:29:37.009700 2015] [core:notice] [pid 17505:tid 140713314006912] AH00094: Command line: '/usr/sbin/apache2
Click to expand...


I have now changed under '/etc/logrotate.d/apache2' following from:
Code: 
...

    postrotate
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 reload > /dev/null; \
                fi;
...

to

Code: 
...
    postrotate
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 restart > /dev/null; \
                fi;
...
And going to look now whether the change solve the problem

What else can I still do?

Thanks in advance
 
Hi omexlu12,

as already suggested, please consider to CHANGE the apache - log - level to "debug" for further investigations and / or :

UFHH01 said:
Another addition afterwards, would be to investigate the root cause of your segfaults or similar errors and you should disable each apache - module ( one after the other ), followed by several apache reloads ( or graceful restarts ).

Example:

a2dismod MODULE_NAME

service apache2 restart
( check that the apache is working as expected, using: "ps -ef | grep apache2" and if yes, please reload apache several times with... )service apache2 reload && service apache2 reload && service apache2 reload && service apache2 reload
( OR: service apache2 graceful && service apache2 graceful && service apache2 graceful && service apache2 graceful )

( check again, if apache is still running as expected. If YES, re-enable the module with "a2enmod MODULE_NAME" and test the next one )​
Click to expand...
​
 
Hi @UFHH01 ,

I have now changed unter '/etc/apache2/apache.conf' to:
'LogLevel debug' and 'service apache2 restart'

I have testet all the modules and do some gracefuls and restarts and there is no problem with all the modules apache is running like a charm here.

So i think there is really some strange problem with the logrotation. Here i have replaced 'reload' by 'restart' as you said and enabled 'LogLevel debug' and now i have to wait until tomorrow to morning 6:30 am if the problem is solved with that and when not we see more informations in errorlog from apache.

Yesterday i have run the daily.cron manually and there no problems occurs only if the cron auto. runs. Maybe give a retry to manuelly run the cron? Maybe with that command:
logrotate -vf /etc/logrotate.conf
Click to expand...
 
Last edited:
Hi omexlu12,

if you want to have decent results for your investigations, you should always consider to investigate issues based on the previous settings, especially if you would like to get to the root cause of a complex issue and changed this or that. So please wait for tomorrow and enjoy your saturday with something else. :)
 
Hi @UFHH01

Yes you are right :) I will wait until tomorrow and lets see what happen, maybe the "fix" with 'restart' instrad of 'reload' broad something , i hope so.
I will keep you noticed tomorrow and maybe with new log entrys in apache log about that :)

EDIT
I read also this post:
http://talk.plesk.com/threads/apach...rt-causes-apache-segfault.335534/#post-790266

I will also change '/etc/logrotate.d/mod_security' from
/var/log/modsec_audit.log {
daily
rotate 7
missingok
compress
postrotate
/etc/init.d/apache2 reload > /dev/null 2>/dev/null || true
endscript
}
Click to expand...
to
/var/log/modsec_audit.log {
daily
rotate 7
missingok
compress
postrotate
/etc/init.d/apache2 restart > /dev/null 2>/dev/null || true
endscript
}
Click to expand...

Maybe here is the same problem...
 
Last edited:
@UFHH01

No Problem this morning after the changes, but i let loglevel debug until tomorrow if then all went good i will reenable nginx as reverse-proxy. :)
 
@UFHH01

After 2 days of tests i think the problem is solved it was definitivly the problem with the crontab of apache and mod_security.
Replacing reload with restart gesolved the issue.

Nginx enabled again all works fine.
Thank you.
 
Last edited:
@omexlu12,

The current solution is not a solution, but a "work-around": instead of reloading, a restart is enforced (i.e. processes are stopped and started, including jammed processes).

Moreover, the "work-around" can become problematic in the future, for two reasons:

- the server does not seem to be able and/or have the resources to "reload" at a specific moment in time, i.e. the time of the cronjobs (approx. 06:45)
- the "restart" approach is a "try again" method, implying that running processes can be endangered at restart time

It should be clear that the root cause of the problem still exists, when applying the current "work-around".

Really, it is not adviceable to apply a "work-around", if the general stability of your system is endangered by the still existing root cause of the problem.

Regards....
 
Hi omexlu12,

there is as well: http://talk.plesk.com/threads/apache-reload-graceful-restart-causes-apache-segfault.335534/

don't pay too much attention on trialotto 's words, because he doesn't know the root case either. There is absolutely nothing wrong to restart apache instead of reloading it and restarting apache won't cause any harm to your server, nor will it harm your system stability. trialotto 's shoot out of the blue is not based on any investigations... unfortunately, he does that quite often the last few weeks, which doesn't lead to anything than to endless discussions, with no solutions. He doesn't even notice, that this sort of discussions are rather frustrating, because they just consume time, but don't bring out any solutions.

I did some research on this issue a few months ago and it is mostly caused on virtual machines - but there are as well dedicated servers, which are affected.
The "combination" of several reloads ( as for example with the standard apache2-logrotation and the mod-security-logrotation, during the daily logrotation process ), is not the root cause. This happens as well on VM's with no mod-security installed. Just google the words "apache" "logrotate" "reload" "segfault".
Actually, this apache - bug can have various root causes, starting from using mod_php, over to conflicts with opcache/xcache and lack of ressources can also be a cause.

... but as I already stated, changing "reload" to "restart" will solve the issue and even trialotto has to admit, that when a work-around solves an issue, further investigations are just time-consuming, when the work-around doesn't produce further issues/failures/problems - which is absolutely not the case.



@trialotto : Consider not to continue another endless discussion, which confuses forum-users, who are trying to solve their issues... and "no"... I'm not going to answer any more hypothetical questions, if they are not based on facts.
 
UFHH01 said:
... but as I already stated, changing "reload" to "restart" will solve the issue and even trialotto has to admit, that when a work-around solves an issue, further investigations are just time-consuming, when the work-around doesn't produce further issues/failures/problems - which is absolutely not the case.
Click to expand...
Using "restart" instead of "reload" causes the server to be unavailable/down for a short time. This is not acceptable for many services. I also see this as a workaround and not solved (Although this might have to be fixed in Debian or somewhere else outside of Plesk scope).
 
omexlu12 said:
It solves the problem and this is fact... :/
Click to expand...
It does not solve the crash on reload - it works around the crash by not using the problematic reload command.

Sure: This somehow solves the problem. It does not happen anymore. But the behavior is not the same.
 
I think your workaround is good for most people. But it's problem for services which can't be offline for a few seconds every night.
I just disabled mod_security as I'm using Cloudflare WAF anyway. Of course this is much more a workaround and not a solution.

Is the modsecurity package from official Debian repositories or Plesk? If it's official Debian the best would be to create a bug report in the Debian bug tracker.
 
Hi Janek,

you are very welcome to trace back the issue, like I already recommended before:

UFHH01 said:
Hi Chris1,

could you please post the output of "apachectl -M" for further investigations?

Do manual gracefull restarts of apache result as well in segfaults? ( Please test the command: "apache2ctl graceful && apache2ctl graceful && apache2ctl graceful && apache2ctl graceful" and have another look at your logs, please ).

Due to the fact that your eMail is from around 05:35, including a gracefull restart at 05:34, without any segfaults and the next gracefull restart of apache was at 05:53 ( WITH the segfault! ), I doubt that the Plesk updates were causing your segfault at all. There must be another cause/reason, why apache was restartet... does your daily/weekly/monthly logrotate starts about this time on your server? If so, please consider to post as well your apache2 - logrotate file ( "/etc/logrotate.d/apache2" ). Consider as well a manual logrotate start with the command "/usr/sbin/logrotate -v -f /etc/logrotate.d/apache2 > /var/log/test_logrotate.log 2>&1".

You could as well capture your segfault, for further investigations, using "strace" for example: First, please stop apache completely and then start it with the command "strace -ff -o /var/log/test_trace.log -r apachectl start" and then try several gracefull restarts and if this doesn't result in a segfault, consider again to use the manual logrotate command above, if this was causing the segfault.
Click to expand...

... so that further investigations can be done!!!


Please note, that the issue doesn't only appear on Debian/Ubuntu - systems, but as well on CentOS/RHEL - based systems and in addition, this issue as well appears on systems without modsecurity installed!


 
Last edited by a moderator:
You must log in or register to reply here.

Similar threads

H
Issue Unable to start apache
Replies
1
Views
1K
Heppi75
H
Oscar Segura
Issue Upgrading from Ubuntu 20.04 to 22.04: failed reboot
Replies
0
Views
1K
Oscar Segura
Oscar Segura
H
Resolved Best way to develop a website in Plesk is by modifying the host file on my Mac
Replies
7
Views
1K
mow
M
A
Issue zend_mm_heap corrupted
Replies
0
Views
965
Ayki
A
Ming
Issue Passenger issue - boost::thread_resource_error: Resource temporarily unavailable
Replies
12
Views
3K
Ming
Ming
Back
Top