VMware vSphere ESXi hostd fails to initialize with the error: vim.fault.AlreadyExists

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• hostd fails to initialize with the error:
  
  vim.fault.AlreadyExists
   
• hostd crashes with a backtrace similar to below

In var/log/hostd.log

==========
2019-08-26T14:57:18.092Z info hostd[2102976] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/57290138-1c0f####-####-########15d/vm_name/vm_name.vmx] Virtual machine object cleanup
2019-08-26T14:57:18.092Z error hostd[2102976] [Originator@6876 sub=Solo.HaVMFolder] Failed to load virtual machine: /vmfs/volumes/57290138-1c0f####-####-########15d/vm_name/vm_name.vmx: Fault cause: vim.fault.AlreadyExists
--> .
2019-08-26T14:57:18.092Z panic hostd[2102976] [Originator@6876 sub=Default]
-->
--> Panic: NOT_IMPLEMENTED bora/vim/hostd/solo/haVmFolder.cpp:318
-->
--> Backtrace:
--> [backtrace begin] product: VMware ESX, version: 6.7.0, build: build-13006603, tag: hostd, cpu: x86_64, os: esx, buildType: release
--> backtrace[00] libvmacore.so[0x00363CAC]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001BC080]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x00360FB8]
--> backtrace[03] libvmacore.so[0x00361072]: Vmacore::PanicExit(char const*)
--> backtrace[04] libvmacore.so[0x0018A6AC]
--> backtrace[05] libvmacore.so[0x0018A74A]
--> backtrace[06] hostd[0x00658E6B]
--> backtrace[07] hostd[0x0065CB53]
--> backtrace[08] libvmacore.so[0x0028D405]
--> backtrace[09] libvmacore.so[0x0028D7E3]
--> backtrace[10] libvmacore.so[0x0036B2FB]
--> backtrace[11] libpthread.so.0[0x00007D3B]
--> backtrace[12] libc.so.6[0x000E9F4D]
--> [backtrace end]

Example: In the /etc/vmware/hostd/vmInventory.xml file.

vmInventory.xml:
====================
<ConfigEntry id="0225">
<objID>257</objID>
<vmxCfgPath>/vmfs/volumes/57290138-1c0f####-####-########15d/vm_name/vm_name.vmx</vmxCfgPath>
</ConfigEntry>
<ConfigEntry id="0226">
<objID>263</objID>
<vmxCfgPath>/vmfs/volumes/57290138-1c0f####-####-########15d/vm_name/vm_name.vmx</vmxCfgPath>
</ConfigEntry>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0

VMware vSphere ESXi 7.0

Cause

This issue occurs due to the duplicate entry for a virtual machine in the /etc/vmware/hostd/vmInventory.xml file.

Resolution

To resolve this issue, remove the duplicate entry for the virtual machine causing this issue.

To remove the duplicate entry for the virtual machine:

1. Connect to the host using SSH. 
2. Navigate to the /etc/vmware/hostd directory using this command:
   
   cd /etc/vmware/hostd
    
3. Open the vmInventory.xml file using a Text Editor.
4. Identify the duplicate entries for the virtual machine in the vmInventory.xml file.
5. Create a backup of the vmInventory.xml file.
6. Comment or remove all the duplicate entries for the VM.
7. Restart hostd by running this command:
   
   /etc/init.d/hostd start