Getting Socket Error 10048 when starting the Policy Server

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

Description:

When the Policy Server starts, it fails with:

[1860/3328][Thu May 05 2011 09:25:17][CServer.cpp:6295][ERROR] Failed to initialize UDP server socket on port: 44444. Socket error:10048

[1860/3328][Thu May 05 2011 09:25:17][CServer.cpp:6373][ERROR] Failed to initialize TCP server socket: Socket error:10048
[1860/3328][Thu May 05 2011 09:25:17][CServer.cpp:6373][ERROR] Failed to initialize TCP server socket: Socket error:10048
[1860/3328][Thu May 05 2011 09:25:17][CServer.cpp:6373][ERROR] Failed to initialize TCP server socket: Socket error:10048
[1860/3328][Thu May 05 2011 09:25:17][CServer.cpp:6373][ERROR] Failed to initialize TCP server socket: Socket error:10048
[1860/3328][Thu May 05 2011 09:25:17][CServer.cpp:5170][ERROR] TCP port initialization failure
[1860/3328][Thu May 05 2011 09:25:17][CServer.cpp:3447][INFO] smpolicysrv shutting down

Solution:

"Socket Error 10048" occurs because the specified address is already in Use. Before starting the policy server, determine the process/service that is using port 44444 and terminate them:

# netstat -an | findstr 44444 on Windows systems
# netstat -an | grep 44444 on Unix/Linux systems

Then terminate the processes holding sockets:

# Using the Task Manager, locate the process "smpolicysrv.exe" and terminate it on Windows systems.
# Using ps and kill commands in Unix/Linux systems

In the smsp.log, you would see the Policy Server starting, but twice:

[3368/620][Thu May 05 2011 09:10:19][CServer.cpp:3004][INFO] Starting up Application: smpolicysrv.
...
[3316/2320][Thu May 05 2011 09:10:58][CServer.cpp:3004][INFO] Starting up Application: smpolicysrv.

See the PID/TID fields [3368/620] and [3316/2320] on both lines, they are different, meaning that 2 different smpolicysrv processes are starting.

The process [3368/620] finishes to initialize:
[3368/620][Thu May 05 2011 09:11:00][CServer.cpp:4438][INFO] SiteMinder Policy Server is ready on PS1_Server

So when the 2nd process [3316/2320] will try to initialized, the socket error 10048 will happen because the port is already used by the first process:

[3316/2320][Thu May 05 2011 09:11:40][CServer.cpp:3359][INFO] Initialized smpolicysrv
[3316/2320][Thu May 05 2011 09:11:40][CServer.cpp:5118][INFO] Initializing TLI
[3316/2320][Thu May 05 2011 09:11:40][CServer.cpp:6310][INFO] Initializing UDP
[3316/2320][Thu May 05 2011 09:11:40][CServer.cpp:6295][ERROR] Failed to initialize UDP server socket on port: 44444. Socket error:10048
[3316/2320][Thu May 05 2011 09:11:40][CServer.cpp:5163][ERROR] UDP port initialization failure
[3316/2320][Thu May 05 2011 09:11:40][CServer.cpp:6373][ERROR] Failed to initialize TCP server socket: Socket error:10048

So the socket error 10048 is a consequence that the smpolicysrv is starting twice with the same parameters, thus trying to open a socket using the same port.
We cannot restrict multiple smpolicysrv processes to run simultaneously (and this may probably be done at OS level anyway) because it should be possible to run several instance of the policy server, when using the -stats flag, for example.

The question here is to know why it was started twice. It happens usually due to operational mistake:

• smpolicysrv being started before it was completely shutdown
  
  
• smpolicysrv being twice started manually
  
  
• smpolicysrv being started manually and by service or a program

Environment

Release:
Component: SMPLC