Web Agent Startup Errors: PID Cache Error, IPC Error, and Shared Memory Allocation Failures

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign-On
• SITEMINDER
• CA Single Sign On Agents (SiteMinder)

Issue/Introduction

When running a Web Agent, and in the Web Server's log, this kind of error appears in the Apache error log:

[14/Apr/2011:14:47:06] [Error] [CA WebAgent IPC][10054][CSmSharedSegment::smalloc] Error allocating shared memory segment using key 0x6cc7b0a7 - File exists (17)
[14/Apr/2011:14:47:06] [Info] [CA WebAgent IPC] [10054][CSmSem::SemRm] Removed semaphore 45285378
[14/Apr/2011:14:47:06] [Error] SiteMinder Agent
    PID Cache error.
    Failed to initialize PID Cache

This issue is most prevalent in Unix/Linux environments running Apache or IBM HTTP Server (IHS) with the SiteMinder Web Agent (all versions).

Cause

SiteMinder Web Agents utilize an architecture where the Web Server process (e.g., Apache child processes) communicates with a long-running supervisor process called LLAWP. This communication relies on System V IPC mechanisms: Shared Memory Segments and Semaphores. 

1. Stale IPC Resources: If a Web Server or LLAWP crashes or is killed (e.g., kill -9), it may leave behind shared memory segments. If a new process tries to create a segment with the same Key but has different owner permissions, the "File exists (17)" or "Permission denied" error occurs.
2. Kernel Limits: The Linux kernel parameters (shmmax, shmall, sem) might be too restrictive for the configured SiteMinder Cache sizes (Resource, and Session caches).
3. Conflict: Multiple Web Agent instances on the same box using the same ServerPath or conflicting IPC keys.

Resolution

1. Identify the Stale Keys (1):
   
   Check the Web Agent error logs (usually WebAgent.log or the Apache error_log) for the specific Hexadecimal or Decimal key ID that failed to allocate.
   
   
2. Manual Cleanup (Critical):
   
   Stop the Web Server: apachectl stop.
   Ensure no LLAWP is running: ps -ef | grep LLAWP. If found, kill -15 <pid>.
   List resources: ipcs -ma.
   Remove resources belonging to the Web Server user:
   ipcrm -m <ID> (for shared memory)
   ipcrm -s <ID> (for semaphores)
   
   
3. Validate Kernel Tuning (2):
   
   Ensure the OS can handle the requested memory. Check /etc/sysctl.conf.
   kernel.shmmax: Should be larger than the largest single shared memory segment needed.
   kernel.sem: Increase if running many Apache child processes.
   
   
4. Testing via Cache Reduction (3):
   
   If the issue persists, set MaxResourceCacheSize, and MaxSessionCacheSize, to 0 in the ACO (Agent Configuration Object). If the Agent starts, the issue is strictly insufficient OS memory resources. 

Additional Information

1. Apache-based Agent Troubleshooting
   
   
2. Operating System Tuning for Agents
   
   
3. Web Agent Caches