nginx 502 error

[omexlu12]

Hi @trialotto ,

For some tests i have now deactivated nginx as reverse-proxy in plesk and use only apache for now. I will test this tomorrow if any other erros appears at this time.
Maybe after this tests i will consider to test out option a with changing the time of the daily cron.

The Problem with option b is that i dont really need a big server or didicated for my little projects, but i never had problems in the past

EDIT: Maybe i will test to run the daily crontask few hours later.

 

[omexlu12]

Hi guys,

Today no problem at all after disabling nginx as reverse proxy, and really no overload of the resources at this time :/
What can cause this 502 at nginx on the crontasks? Is not normal :/

For now i still use only apache as webserver

 

[sdmx80]

Hello forum members!

I am encountering the same problem. A 502 error is displayed minutes after a cronjob of on of my clients is started. This annoying error affects all domains, but it happens only two to three times a week.

Server facts: CentOS 7 (latest updated), Plesk (12.5.30, latest update), Root Server, 16 GB RAM, 2x1 TB on RAID 1

It happend on Monday night and Wednesday night, but only on monday I received a crash report.

:Nov 16 02:30:01 host.example.com CROND[5468]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 02:35:01 host.example.com CROND[5505]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 02:40:01 host.example.com CROND[5552]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 02:45:01 host.example.com CROND[5586]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 02:50:01 host.example.com CROND[5630]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 02:55:01 host.example.com CROND[5671]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 03:00:01 host.example.com CROND[5708]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 03:05:01 host.example.com CROND[6023]: (client) CMD (/opt/plesk/php/5.5/bin/php -f 'httpdocs/website/wp/wp-cron.php')
:Nov 16 03:09:08 host.example.com httpd[8841]: [Mon Nov 16 03:09:08.645756 2015] [so:warn] [pid 8841] AH01574: module actions_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8841]: [Mon Nov 16 03:09:08.651408 2015] [so:warn] [pid 8841] AH01574: module deflate_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8841]: [Mon Nov 16 03:09:08.652093 2015] [so:warn] [pid 8841] AH01574: module headers_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8841]: [Mon Nov 16 03:09:08.652478 2015] [so:warn] [pid 8841] AH01574: module logio_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8841]: [Mon Nov 16 03:09:08.653876 2015] [so:warn] [pid 8841] AH01574: module suexec_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8860]: [Mon Nov 16 03:09:08.877697 2015] [so:warn] [pid 8860] AH01574: module actions_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8860]: [Mon Nov 16 03:09:08.879663 2015] [so:warn] [pid 8860] AH01574: module deflate_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8860]: [Mon Nov 16 03:09:08.880273 2015] [so:warn] [pid 8860] AH01574: module headers_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8860]: [Mon Nov 16 03:09:08.880634 2015] [so:warn] [pid 8860] AH01574: module logio_module is already loaded, skipping
:Nov 16 03:09:08 host.example.com httpd[8860]: [Mon Nov 16 03:09:08.882024 2015] [so:warn] [pid 8860] AH01574: module suexec_module is already loaded, skipping
:Nov 16 03:09:10 host.example.com abrt-hook-ccpp[8861]: Saved core dump of pid 23794 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2015-11-16-03:09:09-23794 (26771456 bytes)
:Nov 16 03:09:10 host.example.com systemd[1]: httpd.service: main process exited, code=dumped, status=11/SEGV
:Nov 16 03:09:10 host.example.com systemd[1]: httpd.service: control process exited, code=exited status=1
:Nov 16 03:09:10 host.example.com systemd[1]: Unit httpd.service entered failed state.

After this report, all websites displayed the 502 gateway error.

How can a simple cronjob cause webservers running Plesk to fail? I think this is a serious issue that needs to be addressed.

Thanks in advance for you support!

- sdmx80

 

[trialotto]

@sdmx80,

In your case, resource overload is caused by a WordPress cronjob.

In that case, two primary factors are relevant:

- the amount of memory assigned to WordPress, AND
- the php5-fpm setting, in specific the "pm" related settings.

Can it be the case that you modified www.conf (in the directory /etc/php5/fpm/pool.d)?

If so, reset the following parameters to the default values:

pm=dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

and that should suffice.

It could be wise to uncomment "pm.max_requests = 500" and set the value to 100 (or something similar), but that really is optional.

If the www.conf has not been altered (i.e. the www.conf is standard), then consider the following two options:

a) if the WordPress memory limit (normally, a custom memory limit would have been specified in the wp-config.php file) has been altered (read: increased):

- reset the WordPress memory limit to a lower value, see explanation below,

AND/OR

b) if the WordPress memory limit has not been altered (i.e. it is standard, at 40MB): go to the domains in question, select PHP settings and apply the following changes:

- increase memory_limit to 256M,
- set max_execution_time and max_input_time to 120 (to be safe, even though these settings are not really relevant in this matter)

and that should do the trick.


The explanation behind the above two options a) and b) is the following.

First of all, option b) allows for memory (assigned to a domain) to be sufficient, in order to execute specific jobs: it is a safeguard that can prevent specific tasks on a domain to jam all running processes, including the cronjob processes.

Second and more important, option a) is

- related to option b), in the sense that increasing the WordPress memory limit only makes sense if the PHP settings are changed accordingly, AND
- relevant for a proper functioning of php-fpm.

To explain the relevance of option a) for php-fpm, note that a pm.max_children=5 actually implies that WordPress, with a default of 40MB, uses up to 200MB of memory.

When having the WP memory limit increased, the memory usage even becomes more.

When (also) having the pm.max_children set to a higher value, it can be the case that multiple GBs of memory are used.

To illustrate, a setting of pm.max_children=400 and a WordPress default memory limit of 40 MB could use up to (more than) 15GB of memory, which causes any small VPS with less than 15GB ram of memory to jam up.

It must by clear by now that one has to align the WordPress (memory) settings and the php-fpm settings, that should be standard by preference.

Hope the above helps!

If any issues still exist after reconfiguration of various settings, just let us know.

Regards....

 

[sdmx80]

@trialotto,

Thank you so much for your detailed explanation! You were spot on!

The parameters in the pool file had much higher values than the standard values. I changed them back to default and will now wait if the error occurs again.

One thing for other people with the same problem: The file you were talking about is not at /etc/php5/fpm/pool.d/www.conf, it is at /etc/php-fpm.d/www.conf.saved_by_psa.

Best regards!

 

[trialotto]

@sdmx80,

My bad, I forgot that you were using CentOS, seems to be an unnecessary typo.

EDIT: it is a typo, indeed. But the www.conf.saved_by_psa is a backup file, not the active one.


Regards....

 

[omexlu12]

Hi @trialotto

Unfortantly the problem is still not solved also i get some problem with apache at the same time :/

Logs:

[Thu Nov 19 06:29:13.264579 2015] [auth_digest:notice] [pid 3428:tid 140696033556352] AH01757: generating secret for digest authentication ...
[Thu Nov 19 06:29:13.440584 2015] [ssl:warn] [pid 3428:tid 140696033556352] AH01909: roundcube.webmail:443:0 server certificate does NOT include an ID which matches the server name
[Thu Nov 19 06:29:13.440894 2015] [ssl:warn] [pid 3428:tid 140696033556352] AH01909: lists:443:0 server certificate does NOT include an ID which matches the server name
[Thu Nov 19 06:29:13.441151 2015] [ssl:warn] [pid 3428:tid 140696033556352] AH01909: default-37_120_175_184:443:0 server certificate does NOT include an ID which matches the server name
[Thu Nov 19 06:29:13.445216 2015] [mpm_event:notice] [pid 3428:tid 140696033556352] AH00489: Apache/2.4.10 (Debian) OpenSSL/1.0.1k Apache mod_fcgid/2.3.9 configured -- resuming normal operations
[Thu Nov 19 06:29:13.445239 2015] [core:notice] [pid 3428:tid 140696033556352] AH00094: Command line: '/usr/sbin/apache2'
[Fri Nov 20 06:28:56.907660 2015] [mpm_event:notice] [pid 3428:tid 140696033556352] AH00493: SIGUSR1 received. Doing graceful restart
[Fri Nov 20 06:28:57.253502 2015] [core:notice] [pid 3428] AH00060: seg fault or similar nasty error detected in the parent process
[Fri Nov 20 06:37:30.003317 2015] [ssl:warn] [pid 19476:tid 140088699189120] AH01909: roundcube.webmail:443:0 server certificate does NOT include an ID which matches the server name
[Fri Nov 20 06:37:30.004982 2015] [ssl:warn] [pid 19476:tid 140088699189120] AH01909: lists:443:0 server certificate does NOT include an ID which matches the server name
[Fri Nov 20 06:37:30.005284 2015] [ssl:warn] [pid 19476:tid 140088699189120] AH01909: default-37_120_175_184:443:0 server certificate does NOT include an ID which matches the server name
[Fri Nov 20 06:37:30.005942 2015] [:notice] [pid 19476:tid 140088699189120] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
[Fri Nov 20 06:37:30.006120 2015] [:notice] [pid 19476:tid 140088699189120] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"
[Fri Nov 20 06:37:30.006130 2015] [:notice] [pid 19476:tid 140088699189120] ModSecurity: PCRE compiled version="8.35 "; loaded version="8.35 2014-04-04"
[Fri Nov 20 06:37:30.006134 2015] [:notice] [pid 19476:tid 140088699189120] ModSecurity: LUA compiled version="Lua 5.1"
[Fri Nov 20 06:37:30.006137 2015] [:notice] [pid 19476:tid 140088699189120] ModSecurity: LIBXML compiled version="2.9.1"
[Fri Nov 20 06:37:30.006140 2015] [:notice] [pid 19476:tid 140088699189120] ModSecurity: Original server signature: Apache
[Fri Nov 20 06:37:30.006143 2015] [:notice] [pid 19476:tid 140088699189120] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Fri Nov 20 06:37:30.006610 2015] [suexec:notice] [pid 19476:tid 140088699189120] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Fri Nov 20 06:37:30.088006 2015] [auth_digest:notice] [pid 19478:tid 140088699189120] AH01757: generating secret for digest authentication ...
[Fri Nov 20 06:37:31.003665 2015] [ssl:warn] [pid 19478:tid 140088699189120] AH01909: roundcube.webmail:443:0 server certificate does NOT include an ID which matches the server name
[Fri Nov 20 06:37:31.005119 2015] [ssl:warn] [pid 19478:tid 140088699189120] AH01909: lists:443:0 server certificate does NOT include an ID which matches the server name
[Fri Nov 20 06:37:31.005394 2015] [ssl:warn] [pid 19478:tid 140088699189120] AH01909: default-37_120_XXX_XXX:443:0 server certificate does NOT include an ID which matches the server name
[Fri Nov 20 06:37:31.009405 2015] [core:warn] [pid 19478:tid 140088699189120] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Nov 20 06:37:31.011066 2015] [mpm_event:notice] [pid 19478:tid 140088699189120] AH00489: Apache/2.4.10 (Debian) OpenSSL/1.0.1k Apache mod_fcgid/2.3.9 configured -- resuming normal operations
[Fri Nov 20 06:37:31.011092 2015] [core:notice] [pid 19478:tid 140088699189120] AH00094: Command line: '/usr/sbin/apache2'

After 'service apache2 restart' all went normal, yesterday all was good so i think its really the cronjob making some problems here.
I will try to set the task (daily) some ours later.

EDIT:
I have now set the daily cron 3 ours later at 9:25 AM lets see if that helps :

25 9 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
#25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

I also have deactived the statistics (awstats) because i dont really need them.
Lets see if that helps.

EDIT2: Yesterday no Problem at 6 am, i hope thats the cronjob ... :/

Maybe something to do with PHP ACPu

 

[omexlu12]

Changing the cronjob brought nothing :/
Now i have the same issue only 3 hours later.

So the problem here is really the cronjob but why, i have testet uptime at this time and all went good i dont unterstand this problem.

Maybe i will give a try to reinstall the server with ubuntu and plesk 12.5 otherwise i will consider to change the server.

But i think the cronjob that cause the problem with apache can only be this:

#!/bin/sh

# run htcacheclean

set -e
set -u

type htcacheclean > /dev/null 2>&1 || exit 0
[ -e /etc/default/apache2 ] || exit 0


# edit /etc/default/apache2 to change this
HTCACHECLEAN_MODE=daemon
HTCACHECLEAN_RUN=auto
HTCACHECLEAN_SIZE=300M
HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk
HTCACHECLEAN_OPTIONS=""

. /etc/default/apache2

[ "$HTCACHECLEAN_MODE" = "cron" ] || exit 0

[ "$HTCACHECLEAN_RUN" = "yes" ] ||
( [ "$HTCACHECLEAN_RUN" = "auto" ] && \
[ -e /etc/apache2/mods-enabled/cache_disk.load ] ) || exit 0

htcacheclean ${HTCACHECLEAN_OPTIONS} \
-p${HTCACHECLEAN_PATH} \
-l${HTCACHECLEAN_SIZE}

@IgorG Is this crontab really needed?

Here again the logs from apache:

[Sat Nov 21 09:29:36.003149 2015] [ssl:warn] [pid 17504:tid 140713314006912] AH01909: roundcube.webmail:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:36.003509 2015] [ssl:warn] [pid 17504:tid 140713314006912] AH01909: lists:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:36.003773 2015] [ssl:warn] [pid 17504:tid 140713314006912] AH01909: default-XXX:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:36.004142 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/) configured.
[Sat Nov 21 09:29:36.004149 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"
[Sat Nov 21 09:29:36.004153 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: PCRE compiled version="8.35 "; loaded version="8.35 2014-04-04"
[Sat Nov 21 09:29:36.004155 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: LUA compiled version="Lua 5.1"
[Sat Nov 21 09:29:36.004157 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: LIBXML compiled version="2.9.1"
[Sat Nov 21 09:29:36.004159 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: Original server signature: Apache
[Sat Nov 21 09:29:36.004161 2015] [:notice] [pid 17504:tid 140713314006912] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Sat Nov 21 09:29:36.004475 2015] [suexec:notice] [pid 17504:tid 140713314006912] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Sat Nov 21 09:29:36.085559 2015] [auth_digest:notice] [pid 17505:tid 140713314006912] AH01757: generating secret for digest authentication ...
[Sat Nov 21 09:29:37.003544 2015] [ssl:warn] [pid 17505:tid 140713314006912] AH01909: roundcube.webmail:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:37.003852 2015] [ssl:warn] [pid 17505:tid 140713314006912] AH01909: lists:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:37.004120 2015] [ssl:warn] [pid 17505:tid 140713314006912] AH01909: default-XXX:443:0 server certificate does NOT include an ID which matches the server name
[Sat Nov 21 09:29:37.007858 2015] [core:warn] [pid 17505:tid 140713314006912] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Sat Nov 21 09:29:37.009678 2015] [mpm_event:notice] [pid 17505:tid 140713314006912] AH00489: Apache/2.4.10 (Debian) OpenSSL/1.0.1k Apache mod_fcgid/2.3.9 configured -- resuming normal operations
[Sat Nov 21 09:29:37.009700 2015] [core:notice] [pid 17505:tid 140713314006912] AH00094: Command line: '/usr/sbin/apache2

Do get it working again i must run 'service apache2 restart'

After googled a bit i have found this issue (other software) but problem the same:
https://i-mscp.net/index.php/Thread/9150-Apache-down-every-morning/

So i think there is a problem with logrotate how can i fix that?

Maybe there is a bug in plesk logratate on debian jessie?

 

[UFHH01]

Hi omexlu12,

as you now already investigated, your issue is caused during logrotation. You should now investigate WHY you get apache segfaults or similar errors, which prevents your webserver to reload or to graceful restart ( please investigate such issues in debug - log - level mode for your apache and try to open another thread, if you need further assistance with the investigations, because your issue has got absolute nothing to do with the thread title "nginx 502 error" ).

In addition, please change any "gracefull" restart ( or "reload" ) to a pure "restart" at all depending logrotation - definitions at "/etc/logrotate.d" - folder and have a look if this solves your issue.

Example: "/etc/logrotate.d/apache2"

Change:

...
    postrotate
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 reload > /dev/null; \
                fi;
...

to :

...
    postrotate
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 restart > /dev/null; \
                fi;
...

Another addition afterwards, would be to investigate the root cause of your segfaults or similar errors and you should disable each apache - module ( one after the other ), followed by several apache reloads ( or graceful restarts ).

Example:

a2dismod MODULE_NAME

service apache2 restart

( check that the apache is working as expected, using: "ps -ef | grep apache2" and if yes, please reload apache several times with... )​

service apache2 reload && service apache2 reload && service apache2 reload && service apache2 reload
( OR: service apache2 graceful && service apache2 graceful && service apache2 graceful && service apache2 graceful )

( check again, if apache is still running as expected. If YES, re-enable the module with "a2enmod MODULE_NAME" and test the next one )

​

Consider as well to include MORE basic informations for your next thread. You didn't even include your operating system, nor did you include your Plesk version ( incl. MU ), nor did you include a list of modules, possible manual changes of your basic configuration for your webserver(s)... and so on. The better your informations are, the better will be possible suggestions and work-arounds from forum-users, willing to help you.

 

[omexlu12]

Hi,

Done. New Thread is now here:
http://talk.plesk.com/threads/logrotation-cause-problems.335926/