SM web agent LLAWP failed to initialized with Apache 2.4.x on startup due to Semaphore issue
Article ID: 5014
Updated On:
Products
Issue/Introduction
The Web Agent fails to initialize the message bus and does not start up properly:
[12/Dec/2016:12:08:50] [Error] SiteMinder Agent
Failed to initialize the message bus.
[12/Dec/2016:12:08:50] [Warning] SiteMinder Agent
SiteMinder agent has encountered initialization errors and will not service requests.
Environment
Policy server = R12.8.7 SPx
WebAgent = r12.52 SP1 CR10
Apache 2.4 - OS all latest version
Cause
When the LLAWP is not shut down properly, typically due to an Apache Restart, issues can occur with the semaphores and shared memory segments on the Web Server:
[12/Dec/2016:12:08:45] [Info] [CA WebAgent IPC] [729] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0xc846175c
[12/Dec/2016:12:08:45] [Info] [CA WebAgent IPC] [729] [CSmSem::getSem] Created semaphore 110952474 using key 0xc846175c
[12/Dec/2016:12:08:45] [Info] [CA WebAgent IPC] [729] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x6b46175c
[12/Dec/2016:12:08:45] [Info] [CA WebAgent LLAWP] [4314] [LLAWP Monitor: LLAWP has been started.]
[12/Dec/2016:12:08:46] [Info] [CA WebAgent IPC] [4315] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x6b46175c
[12/Dec/2016:12:08:46] [Info] [CA WebAgent IPC] [4315] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x6b46175c
[12/Dec/2016:12:08:46] [Info] [CA WebAgent IPC] [4315] [CSmSem::getSem] Created semaphore 110985243 using key 0x6b46175c
[12/Dec/2016:12:08:46] [Error] [CA WebAgent IPC] [4315] [CSmSharedSegment::smalloc] Error allocating shared memory segment using key 0x6c46175c - File exists (17)
[12/Dec/2016:12:08:46] [Info] [CA WebAgent IPC] [4315] [CSmSem::SemRm] Removed semaphore 110985243
[12/Dec/2016:12:08:46] [Error] SiteMinder Agent
PID Cache error.
Failed to initialize PID Cache
[12/Dec/2016:12:08:50] [Info] [CA WebAgent IPC] [729] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x6646175c
[12/Dec/2016:12:08:50] [Error] SiteMinder Agent
Failed to initialize the message bus.
[12/Dec/2016:12:08:50] [Warning] SiteMinder Agent
SiteMinder agent has encountered initialization errors and will not service requests.
[12/Dec/2016:12:08:50] [Info] [CA WebAgent IPC] [729] [CSmSem::SemRm] Removed semaphore 110952474
The above can occur because the Apache Restart process may shut down and start back up faster than the Web Agent's LLAWP process can do so. The LLAWP process handles the semaphore and shared memory segment processes, which handle requests for the Web Agent and Policy Server communication. If the Policy Server and Web Agent are in the process of utilizing these processes, they will not be shut down in a timely manner.
As a result, when the LLAWP process receives the shutdown request, it cannot close these connections quickly enough, so these processes become orphaned (semaphores) or become dropped/non-existent (shared memory segments).
When the Web Server is started back up, the LLAWP process attempts to re-use the existing semaphores, but has no shared memory segments to tie them to, and the LLAWP process cannot initialize the message bus properly, causing the Web Agent to hang, and will not be able to serve requests.
Resolution
Ideally, the best way to avoid the issue is to not use Apache Restarts, and to use the Apache Stop/Start commands to stop the Web Server, allowing some time to pass between these commands, so the LLAWP process can shut down in a timely manner. To resolve the outage, follow the below steps:
1) Stop the web server.
2) Check that LLAWP process and Webserver process has exited completely:
> ps -ef | grep LLAWP
> ps -ef | grep httpd
3) Check for the orphaned semaphores and Shared Memory segments:
- list semaphores:
> ipcs -s
- list shared memory segments:
> ipcs -m
- remove semaphores
> ipcrm -s <semid>
- remove shared memory segments:
> ipcrm -m <shmid>
4) Look in the Web Server error log for the semaphore and shared memory keys.
Those are the ones which need to be removed.
5) After all the semaphores/Shared Memory Segments have been cleared, restart the Webserver/Agent.
6) Also check if the machine has enough space to created semaphores/Shared Memory Segments.
-------------------------------------------
For Step 6: Also check if the machine has enough space to created semaphores/Shared Memory Segments, you simply need to check if the machine you are running the server/Web Agent on has enough Disk Space. I don't anticipate this being an issue, but if you don't have any disk space, or are at a high current disk space capacity, consider freeing some up before you run the process. I am unsure how much is needed, so there isn't a recommended amount. Run a disk space check to determine capacity, and let me know of the current % used.
Feedback
