Max. restarts reached for probe - failed to create server session on port 0, error code 98

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

On a system that had a range of probes on it, some probes fail to restart after a robot restart, and there are the following entries in the probe logs:

nimSession - failed to create server session on port 0, error code 98 

Running a 'netstat -an | grep 480' during the startup may indicate that we are trying to get a port to listen on:

tcp 0 0 #.#.#.#:56846 #.#.#.#:48009 TIME_WAIT 

We will ultimately be unable to get the port for listening on and the probe will go into a failed state.

To verify that the problem is not specific to the probe, one of the probes that is running can be shut down (releasing a port) and one of the problem probes should start up fine now with a free port available. 

Environment

Any UIM/probe Version

Resolution

Try setting the starting probe port in the controller configuration to the last port that a probe was started on.

IM > double click controller > Advanced tab > First probe port number > Apply
or 
AC > robot > probes > click on the ellipsis > configure > setup > First probe port number > Save

Additional Information

logmon example:

Sep 23 14:20:05:977 [140365513185056] logmon: * logmon 4.11 
...
Sep 23 14:20:05:978 [140365513185056] logmon: sockCheckLocalPort - close 9 
Sep 23 14:20:05:979 [140365513185056] logmon: sockCheckLocalPort - close 9 
Sep 23 14:20:05:979 [140365513185056] logmon: sockCheckLocalPort: connect to port 48011 failed 111 (ok - its free) 
Sep 23 14:20:05:979 [140365513185056] logmon: sockServer: next available port is 48011 
Sep 23 14:20:05:979 [140365513185056] logmon: sockServer: bind failed 98 
Sep 23 14:20:05:979 [140365513185056] logmon: nimSession - failed to create server session on port 0, error code 98 
Sep 23 14:20:05:979 [140365513185056] logmon: sockClose:0x26652f0: 
Sep 23 14:20:05:979 [140365513185056] logmon: nimSession failed

Disabling cdm probe allowed the logmon probe to work without any errors.

Workaround was to set the first_probe_port to 48011 since that was the highest port showing in the list of probes on that robot.