Policy Server not starting, seeing Failed to initialize UDP server socket on port: 9999. Socket error:{1}
Article ID: 369568
Updated On:
Products
Issue/Introduction
In this particular case, after patching one Policy Server would no longer startup. The smps log contained these lines:
[CServer.cpp:8702][][][][][][][][][][][][][][][][][][][][][][LogMessage:ERROR:[sm-Server-01650] Failed to initialize UDP server socket on port: 9999. Socket error:{1}][]
[CServer.cpp:7066][][][][][][][][][][][][][][][][][][][][][][LogMessage:ERROR:[sm-Server-03460] UDP port initialization failure][]
Environment
Applicable to Policy Server 12.8.x
Applicable to Linux
Resolution
Typically this type of issue is due to an OS issue with the network stack, or there are zombie/orphaned processes remaining.
The fastest resolution is to restart the server itself to reset all processes.
For troubleshooting, use the commands below to check on the Policy Server process and the ports it is listening to. Also checking for UDP port usage in general.
One note of clarification on this error
Failed to initialize UDP server socket on port: 9999. Socket error
It doesn't mean port 9999 was trying to be used, as the same message appears if there a problem with one of the TCP ports as well.
On a working Policy Server and run these commands as they should all be using the same ports. Here is an example from a test system.
Find the PID, process ID:
# ps -e |grep smp
2796 ? 01:57:15 smpolicysrv
Then find the ports used by the 2796 process:
bash-4.2# netstat -anp | grep 2796
tcp 0 0 ###.###.###.###:37104 ###.###.###.###:6677 ESTABLISHED 2796/smpolicysrv
tcp 0 0 ###.###.###.###:37114 ###.###.###.###:6677 ESTABLISHED 2796/smpolicysrv
tcp 0 0 ###.###.###.###:55402 ###.###.###.###:6677 ESTABLISHED 2796/smpolicysrv
tcp 1 0 ###.###.###.###:60946 ###.###.###.###:6677 CLOSE_WAIT 2796/smpolicysrv
tcp 1 0 ###.###.###.###:60288 ###.###.###.###:2489 CLOSE_WAIT 2796/smpolicysrv
tcp 0 0 ###.###.###.###:37120 ###.###.###.###:6677 ESTABLISHED 2796/smpolicysrv
tcp6 0 0 :::44441 :::* LISTEN 2796/smpolicysrv
tcp6 0 0 :::44442 :::* LISTEN 2796/smpolicysrv
tcp6 0 0 :::44443 :::* LISTEN 2796/smpolicysrv
tcp6 0 0 :::44444 :::* LISTEN 2796/smpolicysrv
tcp6 0 0 ###.###.###.###:44442 ###.###.###.###:41030 ESTABLISHED 2796/smpolicysrv
tcp6 0 0 ###.###.###.###:44442 ###.###.###.###:36290 ESTABLISHED 2796/smpolicysrv
tcp6 0 0 ###.###.###.###:44442 ###.###.###.###:36620 ESTABLISHED 2796/smpolicysrv
udp 0 0 127.0.0.1:55944 0.0.0.0:* 2796/smpolicysrv
udp6 0 0 ::1:43570 :::* 2796/smpolicysrv
udp6 0 0 :::44444 :::* 2796/smpolicysrv
udp6 0 0 :::42108 :::* 2796/smpolicysrv
unix 2 [ ACC ] STREAM LISTENING 319258 2796/smpolicysrv /tmp/.java_pid2796.tmp
unix 2 [ ] DGRAM 35266 2796/smpolicysrv
If you want to see everything listening on UDP, use this command:
bash-4.2# netstat -ltup | grep -i udp
udp 0 0 0.0.0.0:43260 0.0.0.0:* 659/avahi-daemon: r
udp 0 0 localhost:55944 0.0.0.0:* 2796/smpolicysrv
udp 0 0 dmac-128sp7-poli:domain 0.0.0.0:* 2155/dnsmasq
udp 0 0 0.0.0.0:bootps 0.0.0.0:* 2155/dnsmasq
udp 0 0 0.0.0.0:bootpc 0.0.0.0:* 912/dhclient
udp 0 0 0.0.0.0:sunrpc 0.0.0.0:* 639/rpcbind
udp 0 0 localhost:323 0.0.0.0:* 713/chronyd
udp 0 0 0.0.0.0:812 0.0.0.0:* 639/rpcbind
udp 0 0 0.0.0.0:mdns 0.0.0.0:* 659/avahi-daemon: r
udp6 0 0 [::]:dict-lookup [::]:* 2142/dxserver
udp6 0 0 [::]:tsilb [::]:* 2458/dxserver
udp6 0 0 [::]:6677 [::]:* 5672/dxserver
udp6 0 0 localhost:43570 [::]:* 2796/smpolicysrv
udp6 0 0 [::]:44444 [::]:* 2796/smpolicysrv
udp6 0 0 [::]:8289 [::]:* 2597/dxserver
udp6 0 0 [::]:sunrpc [::]:* 639/rpcbind
udp6 0 0 localhost:323 [::]:* 713/chronyd
udp6 0 0 [::]:812 [::]:* 639/rpcbind
udp6 0 0 [::]:42108 [::]:* 2796/smpolicysrv
Feedback
