Issue global 504 error few times a day

[NeoTrafy]

Hello and thanks for the job you're doing, it's been more than 3 years since my last problem but this time I'm not able to solve it by myself.

I'm receiving 504 into all the domains in the server but I'm totally able to login into the plesk panel.

First it started with tons of scoreboard is full...

[Wed Feb 24 04:45:50.502774 2021] [mpm_event:error] [pid 1840:tid 140354641016960] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Feb 24 04:51:39.877061 2021] [mpm_event:error] [pid 1840:tid 140354641016960] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Then I put higher values into the 01-cgi.conf and worked for another hour.

<IfModule mpm_event_module>
LoadModule cgid_module modules/mod_cgid.so
MaxRequestWorkers 1000
ServerLimit 40
</IfModule>

And once again:

[Wed Feb 24 09:19:07.616654 2021] [mpm_event:notice] [pid 29763:tid 140304943200384] AH00489: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed Feb 24 09:19:07.616684 2021] [core:notice] [pid 29763:tid 140304943200384] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Feb 24 09:20:13.623698 2021] [fcgid:warn] [pid 29777:tid 140304943200384] mod_fcgid: process 30289 graceful kill fail, sending SIGKILL
[Wed Feb 24 09:41:47.649635 2021] [mpm_event:notice] [pid 29763:tid 140304943200384] AH00492: caught SIGWINCH, shutting down gracefully

I kept putting higher values into MaxRequestWorkers and Server limit until I reached 2000 and 200 respectively but still is not working properly and after few hours I have to manually restart httpd.

Last thing I tried was to change from MPM event to prefork as I saw in the forums lots of messages saying it solved the problem but it didn't work for me, maybe my values in the prefork are still the default and is not enough. Not even 15 minutes worked when I activated prefork.

It's a bit awkward that it never happened before and since 2 days ago the server is ****ed up.

Could you guys lend my a hand.
Thank you in advance.

 

[Bitpalast]

What you'll need to do is to check you processes for the number of concurrent httpd and php-fpm processes. If there are very many php-fpm processes, analyze why scripts are running so long and frequently that they occupy your web server instances. If there are many httpd instances, you could strace into them to see what they are actually doing. Normally this gives very clear information what websites are causing the many httpd instances.

Frequently such issues arise by bad bots that hammer your websites with useless requests, so once you see the website name either in an strace of by the php-fpm associated file, you can look into their logs to find out more about the cause.

 

[NeoTrafy]

Hey Peter, thanks for the answer.

I've been trying to do some research without success. With ps aux | grep php I see few "php-fpm: pool domain.com". Most of them have 1 or 2 processes, none taking more than that except root with those "php-fpm: master process (/opt/plesk/php/7.4/etc/php-fpm.conf", another for the 7.3, 7.2... etc.

If I check the px aux | grep httpd all I see is +30 processes (I had to restart the httpd few hours ago)

apache 1197 0.1 0.0 2233584 66392 ? Sl 11:21 0:01 /usr/sbin/httpd -DFOREGROUND
apache 2576 0.3 0.0 2233584 71800 ? Sl 11:24 0:02 /usr/sbin/httpd -DFOREGROUND
apache 2914 0.2 0.0 2233584 74872 ? Sl 08:53 0:27 /usr/sbin/httpd -DFOREGROUND
apache 5338 0.2 0.0 2233584 67108 ? Sl 11:28 0:01 /usr/sbin/httpd -DFOREGROUND
apache 7675 0.4 0.0 2233584 64472 ? Sl 11:33 0:01 /usr/sbin/httpd -DFOREGROUND
apache 9298 0.4 0.0 2233584 57916 ? Sl 11:36 0:00 /usr/sbin/httpd -DFOREGROUND

And many more pretty similar. Creating a new one every 3-5 mins. Seems like the process stand there forever and never quits so once the number of processes are equal to the Server limit it just fails and the domains shows the 504 error. But the plesk is still accesible.

I really have no idea to solve this problem.

Greetings.

 

[Bitpalast]

Use
# strace -p <pid>
to look into the httpd processes, so that you can identify the website(s) that is/are doing a lot of things. Then look into their log files. Normally a high number of httpd processes is being caused by bad bots that keep hammering websites with useless requests.