Agent key update happening multiple times a day on Webagent server

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Federation (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• SITEMINDER

Issue/Introduction

We're running a Web Agent and we see the Web Agent receiving many
updates on its Keys, when browsers recieve error 500 :

  [240341/1693812480][Wed May 13 2020
  11:36:18][SmPlugin.cpp:66][INFO][sm-AgentFramework-00170] Agent
  Framework plug-in 'SM_WAF_HTTP_PLUGIN' initialized.  Desription
  'SiteMinder Agent HTTP Plug-in'.

  [240341/1693812480] ADMIN: Successfully processed key update
  attribute 'LAT'.

  [240341/1693812480] ADMIN: Received key update attribute
  'KEY_UPDATE_CURRET'.

  [240341/1693812480] ADMIN: Successfully processed key update
  attribute 'CURENT'.

  [240341/1693812480] ADMIN: Successfully processed key update
  attribute 'NETENT'.

  [240636/1693812480] ADMIN: Received key update attribute
  'KEY_UPDATE_LASTcription 'SiteMinder Agent HTTP Plug-in'.

  [240636/1693812480] ADMIN: Administration Manager initialized.

  [240636/1693812480][Wed May 13 2020
  11:36:44][CSmHighLevelAgent.cpp:191][INFO][sm-AgentFramework-00380]
  HLA: Initialization complete.

We'd like to know :

    - Why is the key change performed multiple times a day even if we
      have enabled static key ?
    - What is triggering the key change ?

Environment

Web Agent 12.52SP1CR10 on Apache 2.4 on RedHat 6;

Resolution

The log line "Received key update attribute" is not about an update,
but rather related to an Apache thread which is starting. The Key
update lines are written when a Apache thread starts.

From the log snippet, the Apache and Web Agent seems to have
resource problem as the Web Agent wrongly writes logs :
    
  The logs you showed present unusual output :

    update attribute 'LAT'
    update attribute 'CURENT'
    update attribute 'NETENT'
    update attribute 'KEY_UPDATE_LASTcription 'SiteMinder Agent HTTP Plug-in'

It seems that both Web Servers have the problem at the same time, and
looking at the Apache error logs, we see that both Apache lost
connection with the backend application servers.

That might indicate a network issue or something related to an
equipment on the network.

If threads are busy to get answer from the backend server, Apache will
need to start further threads to serve the incoming requests, which
will generate KEY_UPDATE lines in the Web Agent logs as the Web Agent
new thread needs to gets its 4 keys and configuration from the Policy
Servedr when starting.

More, check if the Web Server configuration has changed recently if
any tuning as been brought to the threads and process amount the Web
Server should start.