Device Server is already running error when restarting service

Products

• VMware Smart Assurance

Issue/Introduction

• Identifying and killing the Process ID (PID) of process preventing vcmaster service restart
• Smarts NCM is not coming up after vcmaster restart because process is running.
• When third-party agents are running on the server, voyence service reports "Address already in use"
• Voyence service is not running, one or more of ports 9991-9998 are in use.
• Smarts NCM vcmaster service restart fails because process is running. Receive message "Device Server Service is already running"

Message similar to the following is seen when trying to restart the Smarts NCM vcmaster service:

[<USERNAME>@<HOSTNAME> conf]# service voyence restart
Stopping Device Server Service: [ OK ]
Starting Device Server Service: Device Service is already started, stop the other process first before starting the services: Address already in use
[FAILED]

Environment

NCM - 10.1.x

Cause

Restarting the Smarts NCM vcmaster/voyence can sometimes fail to restart the other Smarts NCM services because a few processes are running in backend. When this happens, a message similar to the following will be seen:

[<USERNAME>@<HOSTNAME> conf]# service voyence restart
Stopping Device Server Service: [ OK ]
Starting Device Server Service: Device Service is already started, stop the other process first before starting the services: Address already in use
[FAILED]

Resolution

1. Check the processes that are running using the following commands:
   

   netstat -anp | grep commmgrd | grep LISTEN
   netstat -anp | grep autodiscd | grep LISTEN
   netstat -anp | grep syssyncd | grep LISTEN
   netstat -anp | grep evdispatchd | grep LISTEN
   netstat -anp | grep cfgmgrd | grep LISTEN
   netstat -anp | grep voyenced | grep LISTEN

2. Or use 

   ps -auxx | grep -i NCM

    to find all services that are related to NCM and running in background.
3. Note the PID of any running processes from Step 1 or 2, and kill those processes using the following command:
   

   kill -9 <PID noted from Step1>

4. Restart the vcmaster service as 

   systemctl start vcmaster

   for NCM 10.1.8 and above
   or
   use 

   service vcmaster start

   for rest of the versions like NCM 10.1.6 and below.

Additional Information

The following three processes should not be running and their associated ports should not be used by any other processes:

ps -elf | grep commmgrd
ps -elf | grep autodiscd
ps -elf | grep syssyncd

If any other processes are using these ports, find the associated processes using the following command:

netstat -anp | grep <999>

By default, the voyence service uses ports 9991-9998. Some third-party agents overlap on these ports, preventing the voyence service from starting.

Kill any process has been found to be using the specified ports. Once all processes have been stopped, you can start the service.