CA Access Gateway (SPS) AH00288: scoreboard is full, not at MaxRequestWorkers
Article ID: 130088
Updated On:
Products
CA Single Sign On Secure Proxy Server (SiteMinder)
AXIOMATICS POLICY SERVER
CA Single Sign On SOA Security Manager (SiteMinder)
CA Single Sign-On
Issue/Introduction
We're running a CA Access Gateway (SPS) and this one reports the error :
[Fri Mar 01 09:53:55.553673 2019] [mpm_worker:error] [pid 8594:tid
4152100608] AH00288: scoreboard is full, not at MaxRequestWorkers
and we'd like to know how to fix it ?
[Fri Mar 01 09:53:55.553673 2019] [mpm_worker:error] [pid 8594:tid
4152100608] AH00288: scoreboard is full, not at MaxRequestWorkers
and we'd like to know how to fix it ?
Cause
The error "AH00485: scoreboard is full, not at MaxRequestWorkers" is
known issue and if we read the bug's comments, the symptoms are very
similar to what we experience in your environment. More, we're
investigating the mpm server parameters as we can see in the bug's
comments too. From the bug's comments, we see that there's no work
around or specific configuration to solve this error line and
performance issues when this appear.
So we have big chances to solve this issue by applying the
12.52SP1CR09 to the CA Access Gateway (SPS) and run Apache higher
level of 2.4.25.
Let's highlight some comments here :
Scoreboard full error with event/ssl
A high-traffic web server using event MPM and mostly receiving HTTPS
requests frequently got the error "scoreboard is full, not at
MaxRequestWorkers" and showed very bad performance.
We fixed the issue by reverting from 2.4.2 to 2.2.22, still using
event MPM.
[...]
We've seen AH00485: scoreboard is full, not at MaxRequestWorkers on
2.4.4 with the event MPM, no SSL involved.
[...]
Apache 2.4.10 on Slackware Linux 14.1 x86_64 platform.
I am seeing this about once a minute in the logs:
AH00485: scoreboard is full, not at MaxRequestWorkers
I was able to recover only by a forced restart (stop then start).
[...]
I was able to manage this issue by reducing GracefulShutdownTimeout
value and increasing MaxClients / MaxRequestWorkers value to make
more room for Apache scoreboard .
Also I reduce no of MaxKeepAliveRequests Apache global level.
For more info :- https://www.tectut.com/2016/04/workaround-for-scoreboard-is-full-not-at-maxrequestworkers
[...]
We have successfully used patch in #55 for 50 days now on mid-sized
production server with 1-2 million hits per day. No issues
encountered. Previous issues disappeared (we think the original bug
had been abused in DoS attack, but we might be wrong on this).
[...]
Use all scoreboard entries up to ServerLimit, for 2.4
This looks good. Should be proposed for back port!!
[...]
Fixed in 2.4.25
[...]
In the meantime I've decreased the ServerLimit/ThreadLimit to 5 and
increased the ServerLimit 160 and more. The results with these
settings are very good, no more user complaints (see below).
Otherwise those long running HTTP CONNECT sessions were still maxing
out the total number of allowed processes.
> If you have any more experiences with the patch I am certainly
> interested. Even if it has simply run for some time without (new)
> bugs exposed.
the patch had been deployed to about ~3.000 servers since November
2016 with different work loads from 10 users to 400+ users. After
applying your patch + the ThreadLimit change, there were no more
complaints :)
I've also diffed httpd 2.4.23 + the patch with the version of the
code that landed in 2.4.25 and it's exactly the same. I'm soon going
to roll out 2.4.25 to those boxes.
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
known issue and if we read the bug's comments, the symptoms are very
similar to what we experience in your environment. More, we're
investigating the mpm server parameters as we can see in the bug's
comments too. From the bug's comments, we see that there's no work
around or specific configuration to solve this error line and
performance issues when this appear.
So we have big chances to solve this issue by applying the
12.52SP1CR09 to the CA Access Gateway (SPS) and run Apache higher
level of 2.4.25.
Let's highlight some comments here :
Scoreboard full error with event/ssl
A high-traffic web server using event MPM and mostly receiving HTTPS
requests frequently got the error "scoreboard is full, not at
MaxRequestWorkers" and showed very bad performance.
We fixed the issue by reverting from 2.4.2 to 2.2.22, still using
event MPM.
[...]
We've seen AH00485: scoreboard is full, not at MaxRequestWorkers on
2.4.4 with the event MPM, no SSL involved.
[...]
Apache 2.4.10 on Slackware Linux 14.1 x86_64 platform.
I am seeing this about once a minute in the logs:
AH00485: scoreboard is full, not at MaxRequestWorkers
I was able to recover only by a forced restart (stop then start).
[...]
I was able to manage this issue by reducing GracefulShutdownTimeout
value and increasing MaxClients / MaxRequestWorkers value to make
more room for Apache scoreboard .
Also I reduce no of MaxKeepAliveRequests Apache global level.
For more info :- https://www.tectut.com/2016/04/workaround-for-scoreboard-is-full-not-at-maxrequestworkers
[...]
We have successfully used patch in #55 for 50 days now on mid-sized
production server with 1-2 million hits per day. No issues
encountered. Previous issues disappeared (we think the original bug
had been abused in DoS attack, but we might be wrong on this).
[...]
Use all scoreboard entries up to ServerLimit, for 2.4
This looks good. Should be proposed for back port!!
[...]
Fixed in 2.4.25
[...]
In the meantime I've decreased the ServerLimit/ThreadLimit to 5 and
increased the ServerLimit 160 and more. The results with these
settings are very good, no more user complaints (see below).
Otherwise those long running HTTP CONNECT sessions were still maxing
out the total number of allowed processes.
> If you have any more experiences with the patch I am certainly
> interested. Even if it has simply run for some time without (new)
> bugs exposed.
the patch had been deployed to about ~3.000 servers since November
2016 with different work loads from 10 users to 400+ users. After
applying your patch + the ThreadLimit change, there were no more
complaints :)
I've also diffed httpd 2.4.23 + the patch with the version of the
code that landed in 2.4.25 and it's exactly the same. I'm soon going
to roll out 2.4.25 to those boxes.
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
Environment
4 Policy Server 12.52SP1CR01 on RedHat 6;
2 CA Access Gateway (SPS) 12.52SP01CR01 on RedHat 6;
2 CA Access Gateway (SPS) 12.52SP01CR01 on RedHat 6;
Resolution
You have modified Apache MaxRequestWorkers configuration to work around the
issue.
Upgrade the CA Access Gateway (SPS) to 12.52SP1CR09 will solve the issue.
issue.
Upgrade the CA Access Gateway (SPS) to 12.52SP1CR09 will solve the issue.
Feedback
Yes
No
Powered by
