Error : Noodle_Interupted IOException / Noodle

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On SITEMINDER

Issue/Introduction

CA Access Gateway (SPS) intermittently reports Noodle_Interupted
IOException from SPS logs.

Other times it is seen as Noodle_GenericException in Tomcat
server.log.

Users may see this from a web browser as a result of HTTP error 500.

Environment

Policy server OS: ALL
CA Access Gateway (SPS) : R12.5, R12.51, R12.52, R12.8

Cause

There can be a few different kinds of Noodle exception messages in
SPS, the basic cause might be different but troubleshooting methods
are still the same.

CA Access Gateway (SPS) will rely on backend application server
availability to serve proxy requests.

When the backend application server is not available, or down. By
default, SPS will attempt 3 times to reconnect, if the connections
fail, no "new" connection to the backend server is attempted. Then the
request will result in an error. SPS HttpClient log could help to
narrow down the specific backend application host which was down or
slow response at the time.

Resolution

SPS Configuration change to mitigate:

  Set http_connection_stalecheck="true" from server.conf, and restart
  service, by default this value is false. The stalecheck will ensure
  all the connections in the backend pool are still active, thus when
  request tries a connection, it will avoid rendering a connection
  which is already idled out.

How to avoid unfriendly SPS exception page:

  In server.conf, enable custom error pages in the <customerrorpages>
  block. Set enable to "yes" (the default is no).

  Custom error pages can be defined in SPSErrorMessages.properties and
  WebServerErrorMessages.properties (located in
  $NETE_SPS_ROOT/Tomcat/properties/)

  SPS Admin Guide explains the details under section “Custom Error
  Pages Parameters”.

  #Configurations related to custom error pages

  <customerrorpages>

    #possible values are: "yes", "no"
    #default value is "no"
    enable="yes"

     [...]

     locale_country="US"

   </customerrorpages>

  #Custom error pages configuration end

Additional SPS Tuning:

If for some reason enabling stalecheck introduces performance delay,
then additional SPS tuning may be required. In fact, any new SPS
setup, before going live, should go through a tuning exercise first.

Before doing that, please check proxyrules.xml file, ensure SPS does
not forward requests back to SPS host itself under default rule. This
is a common mis-configuration that does have dire consequences for
performance. This can create a looping situation which quickly chews
up all the available connections/threads.

Once that is checked, move on the other tuning options:

  a) ajp13.max_threads=100 (should be slightly > httpd worker
     threads). It is expected (in normal proxy environment) that the
     majority of the time, SPS will be waiting for a response from the
     backend server, so for best throughput it is important that the
     pipe "width" through each component of SPS has a size that does
     not force most to wait for a free "thread" or "connection".

     For best throughput max_threads needs to be enough worker threads to
     service ALL Apache threads. So this value should be slightly greater
     than Apache worker threads. Apache worker threads are set in
     httpd-mpm.conf as :

     MaxRequestWorkers 400 (for worker mode/unix) ThreadsPerChild 150 (for
     windows).

     Suggest default of 410 greater than worker mode max.

     ajp13.max_threads=410

  b) http_connection_pool_max_size="20" (should be > max-threads) As
     described previously, for best throughput there needs to be a
     spare backend socket for each worker thread. So connection max
     pool size should be slightly greater than max_thread
     setting. Assuming max_thread setting of 410 from prior example,
     then suggest set :

     http_connection_pool_max_size="420"

     The back end server should be configured to handle the load of
     such thread pool size as well.

  c) http_connection_timeout="0" (should not be zero). Suggest make
     it some moderately large number rather than wait forever.

     http_connection_timeout="3 minutes"

     http_connection_pool_connection_timeout="1 minute"

Additional Information

  CA Access Gateway (SPS) http_connection_stalecheck="true" recommended
  https://knowledge.broadcom.com/external/article?articleId=43343

  Error : Noddle_GenericException Intermittent CA Access Gateway (SPS)
  https://knowledge.broadcom.com/external/article?articleId=48277

  CA Access Gateway (SPS) : Commonly Tuned Parameters
  https://knowledge.broadcom.com/external/article?articleId=43275

DISCLAIMER: All performance tuning should be bench tested and verified
by client themselves in QA environment before rolling out to
production, and due to system setup variations, client may experience
different test outcome. If additional help is required, Broadcom
Services can be contacted on a contract basis. In no event shall
Broadcom be liable for any loss or damage of any kind, including but
not limited to, special, indirect, punitive incidental or
consequential damage, arising out of the delivery, use, evaluation,
support or operations provided even if Broadcom has advised clients of
the possibility of such damage.