Resolved AH00485: scoreboard is full, not at MaxRequestWorkers

[solucionesuno]

Apache hangs randoly with this messages:

[Mon Feb 27 23:45:28.802339 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:29.803414 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:30.804495 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:31.805404 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:32.806512 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:33.807586 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:34.808669 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:35.809781 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:36.810869 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:37.811959 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers
[Mon Feb 27 23:45:38.813022 2017] [mpm_event:error] [pid 21941:tid 140620660758656] AH00485: scoreboard is full, not at MaxRequestWorkers

i have this configuration:

<IfModule mpm_event_module>
    KeepAlive On
    KeepAliveTimeout 5
    MaxKeepAliveRequests 128

    ServerLimit 10
    StartServers 4
    ThreadLimit 128
    ThreadsPerChild 128
    MinSpareThreads 256
    MaxSpareThreads 512
    MaxRequestWorkers 1280
    MaxConnectionsPerChild 2048
</IfModule>

Server version: Apache/2.4.6 (CentOS)
Server built:   Nov 14 2016 18:04:44
Server's Module Magic Number: 20120211:24
Server loaded:  APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

I read in some formus that its a bug of Apache
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555


Any ideas why?

 

[UFHH01]

Hi solucionesuno,

I read in some formus that its a bug of Apache
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555


Any ideas why?

Pls. keep in mind, that a used software on your server might have bugs, that's why it is recommended to update/upgrade the software packages on a regulary basis on your server. As you can see within your link, this apache - bug has been "RESOLVED FIXED" in version "2.4.7". Before you update/upgrade your server software, you should consider to visit => https://docs.plesk.com/release-notes/onyx/software-requirements/ , to make sure, that the next aupdated/upgraded version is as well supported by Plesk, to avoid issues/errors/problems.

Now back to your question: Any ideas why?
The answer WHY an installed software is buggy, is mostly discussed in the depending forum(s) and going through you mentioned bug - report/discussion, there are several possible issues, why this could happen. The participating users mention slow connections, big files and hung processes during gracefull restarts.
I would recommend to split large files in your webhosting and recommend as well to NOT TO USE MPM EVENT, but MPM PREFORK, in combination with PHP-FPM ( with at least PHP 5.6 ).

 

[solucionesuno]

Hi solucionesuno,


Pls. keep in mind, that a used software on your server might have bugs, that's why it is recommended to update/upgrade the software packages on a regulary basis on your server. As you can see within your link, this apache - bug has been "RESOLVED FIXED" in version "2.4.7". Before you update/upgrade your server software, you should consider to visit => https://docs.plesk.com/release-notes/onyx/software-requirements/ , to make sure, that the next aupdated/upgraded version is as well supported by Plesk, to avoid issues/errors/problems.

Thnaks for reply

I have up to date the plesk and system, and in any update, suggest update Apache to 2.4.7.

yum check-updates
Complementos cargados:fastestmirror
PLESK_17_0_17-extras                                                                                                                                                                                                  | 2.9 kB  00:00:00
PLESK_17_NGINX                                                                                                                                                                                                        | 2.9 kB  00:00:00
PLESK_17_PHP56                                                                                                                                                                                                        | 2.9 kB  00:00:00
PLESK_17_PHP70                                                                                                                                                                                                        | 2.9 kB  00:00:00
base                                                                                                                                                                                                                  | 3.6 kB  00:00:00
epel/x86_64/metalink                                                                                                                                                                                                  |  21 kB  00:00:00
epel                                                                                                                                                                                                                  | 4.3 kB  00:00:00
extras                                                                                                                                                                                                                | 3.4 kB  00:00:00
plesk-letsencrypt                                                                                                                                                                                                     | 2.9 kB  00:00:00
plesk-letsencrypt-tp                                                                                                                                                                                                  | 2.9 kB  00:00:00
plesk-migrator                                                                                                                                                                                                        | 2.9 kB  00:00:00
plesk-migrator-tp                                                                                                                                                                                                     | 2.9 kB  00:00:00
updates                                                                                                                                                                                                               | 3.4 kB  00:00:00
(1/2): epel/x86_64/updateinfo                                                                                                                                                                                         | 748 kB  00:00:00
(2/2): epel/x86_64/primary_db                                                                                                                                                                                         | 4.6 MB  00:00:00
Loading mirror speeds from cached hostfile
 * base: centos.mirrors.ovh.net
 * epel: epel.mirrors.ovh.net
 * extras: centos.mirrors.ovh.net
 * updates: centos.mirrors.ovh.net

kernel-headers.x86_64                                                                                               3.10.0-514.6.2.el7                                                                                                updates
log4cplus.x86_64                                                                                                    1.1.3-0.4.rc3.el7                                                                                                 epel
microcode_ctl.x86_64                                                                                                2:2.1-16.1.el7_3                                                                                                  updates
perl-JSON-XS.x86_64                                                                                                 1:3.01-2.el7                                                                                                      epel
php-imap.x86_64                                                                                                     5.4.16-7.el7                                                                                                      epel
python-perf.x86_64                                                                                                  3.10.0-514.6.2.el7                                                                                                updates

Perhaps a don´t have installed de correct repos, to update apache automatically?

 

[solucionesuno]

I would recommend to split large files in your webhosting and recommend as well to NOT TO USE MPM EVENT, but MPM PREFORK, in combination with PHP-FPM ( with at least PHP 5.6 ).

What do you mean with split large files? what is concider an large files?-

What about MPM_PREFORK vs MPM_EVENT performance? are similar? Mpm_event was the default apache configuration in my plesk installation, so I figured this was the best.

 

[AYamshanov]

hmmm... I too saw ver.2.4.7. So I checked it on my test fresh and updated server and the version of httpd is the same.

[root@centos-512mb-lon1-01 ~]# yum update | tail -n 1
No packages marked for update

[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

[root@localhost ~]# httpd -V
Server version: Apache/2.4.6 (CentOS)
Server built: Nov 14 2016 18:04:44
Server's Module Magic Number: 20120211:24
Server loaded: APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2

 

[UFHH01]

Hi solucionesuno,

on some operating systems ( like for example CentOS 7 ), you might recognize, that actual software changes are backported, instead of upgraded to a newer version. Pls. consider to research at "centos.org", which recent bugs are resolved / updated / upgraded for your apache - webserver on the official CentOS - repos. ( from my point of view, they are confusing users, but this is only MY opinion! ^^ )

There are a lot of reasons to use MPM prefork, instead of MPM event, but to be honest, this is not a trivial discussion and depends as well on your server and domain configuration and modifications and last but not least on your content. Such a discussion should be done with apache - specialists, or with people, who have a lot of time experimenting and researching and depending log - investigations. I think, that the Plesk Community forum is not the right place for such a discussion.
Pls. consider to experiment with the two versions, in combination with different PHP - versions and handlers, so you can be sure, that the "best" choice depends on your very own specific hosting content. Even that the MPM event might be the "first" choice, when you installed the server, it doesn't mean, that this choice is the "best" for YOUR hosting content on your server.

 

[UFHH01]

Hi solucionesuno,

What do you mean with split large files? what is concider an large files?-

"normally" ... large files could be declared from "500 MB upwards" ... but again, this is ONLY a recommendation and no guideline. It really depends on your webhosting content and last but not least on your daily visitors as well. Tweaking your server, the hardware, the software and the content is a very unique process and can't be answered with one or two sentences. - I'm sorry that I can't give you a "better", more satisfactory response.

 

[solucionesuno]

Hello

I can check that one site in my plesk was infected. I can find it with an script like this:

#!/bin/sh


len=`cat /var/log/httpd/error_log |grep Max |wc -l`

if [ $len -gt 1 ]; then

        #!/bin/bash
        # script to send simple email
        # email subject

        SUBJECT="AH00485: scoreboard is full, not at MaxRequestWorkers"

        # Email To ?
        EMAIL="[email protected]"

        # Email text/message
        EMAILMESSAGE="/tmp/emailmessage.txt"
        echo "AH00485: scoreboard is full, not at MaxRequestWorkers" > $EMAILMESSAGE

        links -dump http://localhost:8001/server-status  >> $EMAILMESSAGE


        echo "##################################################################" >> $EMAILMESSAGE



        # send an email using /bin/mail
        /bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE
fi

in this way i find that one site in the moment that Scoreboards is full begins, i get by email an server status.

Resolving this maleware in php file, problem solves.

Regards.

 

[Enverex]

bug has been "RESOLVED FIXED" in version "2.4.7".

Just wanted to point out for anyone else finding this thread that UFHH01 is misreading the bug-report table. 2.4.7 in this instance is the version that the bug was logged against, NOT the version that the issue was fixed in. It was actually fixed in 2.4.25.

 

[Goldecker]

My Apache Version ist 2.4.29 and I get the same message. The Server was running a few years without restarting anything. For a couple of weeks I got the error "server reached MaxRequestWorkers". I set MaxRequestWorkers to 1000.
Now I get "scoreboard is full, not at MaxRequestWorkers. Increase ServerLimit".

 

[websavers]

Just wanted to point out for anyone else finding this thread that UFHH01 is misreading the bug-report table. 2.4.7 in this instance is the version that the bug was logged against, NOT the version that the issue was fixed in. It was actually fixed in 2.4.25.

And as far as I can tell the patch was not backported to 2.4.6 on CentOS 7, so the issue will remain on CentOS 7 boxes.

Setting MaxConnectionsPerChild to 0 as described here, helps to delay the issue as graceful restarts trigger less process kill signalling + forking. On a busy server increasing the total number of processes and setting MaxConnectionsPerChild to 0 together took us from multiple events per day to one every week or so, which isn't so bad, especially if Watchdog catches the outage and restarts httpd with a non-graceful trigger. However if you don't have Watchdog enabled, you may wish to also set a cron job that restarts apache (not-graceful) perhaps once a week or at whatever interval best works for your server's configuration.

 

[websavers]

My Apache Version ist 2.4.29 and I get the same message. The Server was running a few years without restarting anything. For a couple of weeks I got the error "server reached MaxRequestWorkers". I set MaxRequestWorkers to 1000.
Now I get "scoreboard is full, not at MaxRequestWorkers. Increase ServerLimit".

It is still possible under normal conditions for this to happen in a manner that isn't related to that bug. Here's the scenario that seems to be most common:

1. A site takes on hundreds of requests that are passed through apache to PHP-FPM (like uncached WordPress page requests)
2. Each of those requests spin (typically due to either limited resources or a bug in the PHP code causing a spinlock) until they reach their PHP timeout. Let's say that's 5 minutes. This ties up the apache processes that passed the initial request through to PHP-FPM
3. An apache graceful restart signals to its processes to die, but because they're busy serving requests, they say 'nope! Gotta wait to hear back before I die'
4. And now we're in a similar position as that bug presents, but with legitimate reason

IMO there should be an override for this when waiting on an upstream process (like PHP-FPM) but as many have pointed out, the true fix here is to limit your PHP timeouts or set your PHP limits better so the site itself isn't overwhelmed like that.

One workaround is to configure the domain for nginx direct to PHP-FPM and bypass apache entirely for the site experiencing traffic that blocks the apache processes from dying (note the security implications if you're using mod_sec through apache). Another is to severely limit the PHP-FPM timeout values for the site.

I'd be curious if someone has other ideas for how to handle this.