CA Access Gateway (SPS) AH00288: scoreboard is full, not at MaxRequestWorkers

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On
• CA Single Sign On Agents (SiteMinder)
• SITEMINDER

Issue/Introduction

 

When running a CA Access Gateway (SPS), this one reports the error :

  [Fri Mar 01 09:53:55.553673 2019] [mpm_worker:error]
  [pid 8594:tid 4152100608]
  AH00288: scoreboard is full, not at MaxRequestWorkers 

 

Cause

 

The error is a known bug in Apache, which is solved in Apache 2.4.25
(1).

The error "AH00485: scoreboard is full, not at MaxRequestWorkers" is
known issue and reading the bug's comments, there's no work around or
specific configuration to solve this error line and performance issues
when this appears (2).

The CA Access Gateway (SPS) 12.52SP1CR01 run Apache 2.4.4 (3).

The CA Access Gateway (SPS) 12.52SP1CR09 runs Apache 2.4.33 which will
solve this issue (4).

Please note that Policy Server 12.52SP1CR01 is Out of Support (5).

 

Environment

 

Policy Server 12.52SP1CR01 on RedHat 6; 
CA Access Gateway (SPS) 12.52SP01CR01 on RedHat 6; 

Resolution

 

Upgrade the CA Access Gateway (SPS) to the latest version in order 
to get the Apache running on 2.4.33.

 

Additional Information

 

(1)

    Bug 53555 - Scoreboard full error with event/ssl 

      A high-traffic web server using event MPM and mostly receiving HTTPS 
      requests frequently got the error "scoreboard is full, not at 
      MaxRequestWorkers" and showed very bad performance. 

      [...] 

      Fixed in 2.4.25 

    https://bz.apache.org/bugzilla/show_bug.cgi?id=53555 

(2)

    Scoreboard full error with event/ssl 

      A high-traffic web server using event MPM and mostly receiving HTTPS 
      requests frequently got the error "scoreboard is full, not at 
      MaxRequestWorkers" and showed very bad performance. 

      We fixed the issue by reverting from 2.4.2 to 2.2.22, still using 
      event MPM. 

      [...] 

      We've seen AH00485: scoreboard is full, not at MaxRequestWorkers on 
      2.4.4 with the event MPM, no SSL involved. 

      [...] 

      Apache 2.4.10 on Slackware Linux 14.1 x86_64 platform. 

      I am seeing this about once a minute in the logs: 
      AH00485: scoreboard is full, not at MaxRequestWorkers 

      I was able to recover only by a forced restart (stop then start). 

      [...] 

      I was able to manage this issue by reducing GracefulShutdownTimeout 
      value and increasing MaxClients / MaxRequestWorkers value to make 
      more room for Apache scoreboard . 

      Also I reduce no of MaxKeepAliveRequests Apache global level. 

      For more info :- https://www.tectut.com/2016/04/workaround-for-scoreboard-is-full-not-at-maxrequestworkers 

      [...] 

      We have successfully used patch in #55 for 50 days now on mid-sized 
      production server with 1-2 million hits per day. No issues 
      encountered. Previous issues disappeared (we think the original bug 
      had been abused in DoS attack, but we might be wrong on this). 

      [...] 

      Use all scoreboard entries up to ServerLimit, for 2.4 

      This looks good. Should be proposed for back port!! 

      [...] 

      Fixed in 2.4.25 

      [...] 

      In the meantime I've decreased the ServerLimit/ThreadLimit to 5 and 
      increased the ServerLimit 160 and more. The results with these 
      settings are very good, no more user complaints (see below). 

      Otherwise those long running HTTP CONNECT sessions were still maxing 
      out the total number of allowed processes. 

      > If you have any more experiences with the patch I am certainly 
      > interested. Even if it has simply run for some time without (new) 
      > bugs exposed. 

      the patch had been deployed to about ~3.000 servers since November 
      2016 with different work loads from 10 users to 400+ users. After 
      applying your patch + the ThreadLimit change, there were no more 
      complaints :) 

      I've also diffed httpd 2.4.23 + the patch with the version of the 
      code that landed in 2.4.25 and it's exactly the same. I'm soon going 
      to roll out 2.4.25 to those boxes. 

    https://bz.apache.org/bugzilla/show_bug.cgi?id=53555 
        

(3)

    Third-Party Software Acknowledgments 

      Apache HTTP Web Server 2.4.4 

    https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-52-01/third-party-software-acknowledgments.html

(4)   

    CA Access Gateway 

      | SalesForce | Internal | Issue Description                      |
      | Case       | Defect   |                                        |
      | Number     | ID       |                                        |
      |------------+----------+----------------------------------------|
      | 00949107   | DE342765 | OpenSSL is upgraded to OpenSSL 1.0.2o. |
      |            |          | Apache is upgraded to Apache 2.4.33.   |

    https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-52-01/release-notes/cumulative-releases/defects-fixed-in-12-52-sp1-cr09.html#DefectsFixedin12.52SP1CR09-smsps

(5)

    CA Single Sign-On (formerly CA SiteMinder) Release and Support
    Lifecycle Dates

      12.52 February 28, 2019 - EOS 

    https://support.broadcom.com/external/content/release-announcements/CA-Single-Sign-On-Release-and-Support-Lifecycle-Dates/6562