Virtual machines appear to be running or registered on multiple ESX/ESXi Servers
search cancel

Virtual machines appear to be running or registered on multiple ESX/ESXi Servers

book

Article ID: 338070

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • Virtual machine appears to be running or registered on multiple ESX/ESXi hosts:
  • This issue occurs when:
     
    • A vMotion fails to complete correctly or times out in VirtualCenter.
    • A DRS issue where virtual machines are migrated by vMotion automatically in quick succession.
    • When a machine is powered on during VMware HA failover.
    • The Service Console on an ESX/ESXi host is low on memory starving the vpxa process.
       
  • In vCenter Server, you see the virtual machine as appearing on one ESX/ESXi host for a few seconds, then it seems to be on the other.
  • The virtual machine may appear to jump back and forth among different ESX/ESXi hosts.


Environment

VMware ESXi 4.1.x Installable
VMware vCenter Server 5.0.x
VMware ESXi 3.5.x Installable
VMware vCenter Server 5.1.x
VMware vCenter Server Appliance 5.5.x
VMware vCenter Server Appliance 5.1.x
VMware vCenter Server 6.0.x
VMware ESXi 4.0.x Embedded
VMware ESX 4.1.x
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.5
VMware VirtualCenter 2.5.x
VMware vCenter Server 4.1.x
VMware vCenter Server Appliance 6.0.x
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Embedded
VMware ESX Server 3.5.x
VMware ESX Server 3.0.x
VMware vSphere ESXi 6.0
VMware vCenter Server 4.0.x
VMware vCenter Server 5.5.x
VMware ESX 4.0.x
VMware vCenter Server Appliance 5.0.x
VMware VirtualCenter 2.0.x

Cause

This issue occurs when the virtual machine is registered on multiple ESX/ESXi hosts at the same time.

Resolution

ESX:

 
To correct this misconfiguration:
  1. Click Inventory in the navigation bar.
  2. Expand the inventory as needed and click the appropriate managed host.
  3. Click the Virtual Machines tab.
  4. Note the virtual machine that disappears every few seconds.
  5. Log in as root with SSH to both affected ESX hosts.
  6. Run the vmware-cmd -l command to display the names of the virtual machines registered on this host.

    Run the vm-support -x command to show which virtual machines are currently running on the ESX host.

    Compare results from these commands to determine the ESX host on which the virtual machine is registered, but not running. When you have determined this, you need to unregister the virtual machine from the ESX host on which it is registered but not running.
     
  7. Run this command to unregister the virtual machine from the ESX host:

    vmware-cmd -s unregister /vmfs/volumes/datastore/folder/machine.vmx
     
  8. If the virtual machine has a process (PID) associated with it, ESX host may not allow you to unregister it and the command fails with the error:

    VMControl error -999: Unknown error: SoapError: ServerFaultCode(0): (The attempted operation cannot be performed in the current state (Powered On).)

    If you see this error and are unable to unregister the virtual machine:
     
    • Ensure the virtual machine is powered off. For more information, see Powering off an unresponsive virtual machine on an ESX host (1004340).
    • Unregister the virtual machine from the ESX host again with the command:

      vmware-cmd -s unregister </vmfs/volumes/datastore/folder/machine>.vmx
       
    • Run this command to stop the hostd process:

      service mgmt-vmware stop
       
    • Use a text editor to open the /etc/vmware/hostd/vmInventory.xml file.
    • Locate the virtual machine you want to remove.
    • Remove all the information between the <ConfigEntry> tags for the affected virtual machine.
    • Run this command to start the hostd process:

      service mgmt-vmware start
     
  9. Log in to all the ESX host directly using vSphere Client.

    You see the virtual machine on both ESX hosts with a Powered-on status. One host however does not display any details of VMware-Tools, the guest's IP address, and other pertinent information typically displayed in the Summary tab.
  10. Click the virtual machine on the host that does not display any details in the Summary tab.
  11. Right-click the virtual machine and click Power Off.
Note: VMware recommends restarting the mgmt-vmware and vmware-vpxa processes on any hosts on which you have changed registered machines from the command line. For more information, see Restarting the Management agents on an ESX Server (1003490).
 

ESXi:

 
To correct this misconfiguration:
 
  1. Click Inventory in the navigation bar.
  2. Expand the inventory as needed and click the appropriate managed host.
  3. Click the Virtual Machines tab.
  4. Note the virtual machine that disappears every few seconds.
  5. Log in as root with SSH to both affected ESXi hosts.
  6. Run the vim-cmd vmsvc/getallvms command to display the names of the virtual machines registered on this host.

    Run the esxtop and then press shift + V keys to show which virtual machines are currently running on the ESXi host.

    Compare results from these commands to determine the ESXi host on which the virtual machine is registered, but not running. When you have determined this, you need to unregister the virtual machine from the ESXi host on which it is registered but not running.
     
  7. Run this command to unregister the virtual machine from the ESXi host:

    vim-cmd vmsvc/unregister vmid
     
  8. If the virtual machine has a process (PID) associated with it, ESXi host may not allow you to unregister it and the command fails.

    If you are unable to unregister the virtual machine:
    1. Ensure the virtual machine is fully powered off.
    2. Unregister the virtual machine from the ESX host again with the command:

      vim-cmd vmsvc/unregister vmid
       
    3. Run this command to stop the hostd process:

      /etc/init.d/hostd stop
       
    4. Use a text editor to open the /etc/vmware/hostd/vmInventory.xml file.
    5. Locate the virtual machine you want to remove.
    6. Remove all the information between the <ConfigEntry> tags for the affected virtual machine.
    7. Run this command to start the hostd process:

      /etc/init.d/hostd start
  9. Log in to all the ESXi host directly using vSphere Client.

    You see the virtual machine on both ESXi hosts with a Powered-on status. One host however does not display any details of VMware-Tools, the guest's IP address, and other pertinent information typically displayed in the Summary tab.
     
  10. Click the virtual machine on the host that does not display any details in the Summary tab.
  11. Right-click the virtual machine and click Power Off.
Note: VMware recommends restarting the services.sh restart and /etc/init.d/vpxa restart processes on any hosts on which you have changed registered machines from the command line. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).
 
If your problem persists even after trying the steps in this article, file a support request with VMware Support and note this Knowledge Base article ID (1005051) in the problem description.

Additional Information

Powered by Wolken Software