VMware ESX Server Host Agent is already running

Products

• VMware vSphere ESXi

Issue/Introduction

The mgmt-vmware service is the service that starts hostd. Instead of starting hostd directly, mgmt-vmware runs a script called vmware-watchdog which in turn starts hostd as a child process. After vmware-watchdog starts hostd, it waits for (monitors) it and restarts hostd if hostd fails or generates debug/support info after a number of failures.

When mgmt-vmware starts successfully and hostd is running normally, ps shows the following processes:

The watchdog (parent process):

/bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u

The hostd (child process):

/usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u

Before mgmt-vmware tells vmware-watchdog to start hostd, it first tells vmware-watchdog to check if hostd is already running. mgmt-vmware does not tell vmware-watchdog to start hostd if vmware-watchdog reports that hostd is already running.

vmware-watchdog performs the following checks to see if hostd is already running:

1. Is there something in the /var/run/vmware/watchdog-hostd.PID file? When vmware-watchdog starts hostd, it writes the PID of the watchdog process to this file.
2. Is there a vmware-watchdog hostd monitoring process already running? It checks the ps output.

When mgmt-vmware service or hostd is stopped, vmware-watchdog deletes the watchdog-hostd.PID file and exits. Sometimes, if hostd or vmware-watchdog fails to exit cleanly, hostd terminates but vmware-watchdog may still be running or may not delete the watchdog-hostd.PID, therefore preventing the mgmt-vmware service from starting again.

Symptoms:

• The mgmt-vmware service does not start because it incorrectly thinks that the Host Agent (hostd) is already running.
• The following dialog appears when trying to start mgmt-vmware:
  
  VMware ESX Server Host Agent is already running
  
  
• Listing the running processes (ps aux) shows that hostd is not running.

Environment

VMware ESX Server 3.5.x
VMware ESX Server 3.0.x

Resolution

Note: Before you begin, refer to Restarting the Management agents on an ESX Server (1003490) for important information on restarting the mgmt-vmware service.

This fix only applies if hostd is not running.

Run the following command to confirm that hostd is not running:

ps -auxwww | grep vmware-hostd | egrep -v "watchdog|grep"

If hostd is not running, but mgmt-vmware (vmware-watchdog) thinks it is running:

1. Run the following commands to stop the vmware-watchdog hostd monitoring process:
   
2. • ps -auxwww | grep "watchdog -s hostd" | grep -v grep
   • kill <PID returned from above command>
     
     
3. Run the following command to delete the watchdog-hostd.PID file:
   
   rm /var/run/vmware/watchdog-hostd.PID
   
   
4. Run the following command to start hostd:
   
   service mgmt-vmware start

Additional Information

To check if vmware-watchdog is monitoring a process you can use a command of the form: vmware-watchdog -r <tag>

# vmware-watchdog -r hostd

To stop a vmware-watchdog monitor process (does not stop the process being monitored) use a command of the form: vmware-watchdog -k <tag>

# vmware-watchdog -k hostd

Note: If the PID number stored in /var/run/vmware/watchdog-hostd.PID is not running, this command fails and does not delete the PID file. The file must then be deleted manually.

Note: This information can be applied to any process started by vmware-watchdog, including VPXA (vmware-vpxa service).