Virtual machines fail to start with timeout error before shutting down many virtual machines

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Virtual machine fails to power on with the following error:
  Cannot power on [Virtual machine name] on [ESXi host name] in [data center name]. Operation timed out.
   
• The failed virtual machines became invalid status in vCenter Server inventory list.
  Over 30 shutdown tasks for virtual machines has been operated at the same time before powering on virtual machines.
   
• In the /var/log/hostd.log file, you see entries similar to:
  XXXX-XX-XXTXX:XX:XX.XXXZ info hostd[7D340B70] [Originator@6876 sub=Vmsvc.vm:[vmx file path]]
  Error encountered while retrieving configuration. Marking configuration as invalid: N3Vim5Fault20GenericVmConfigFault9ExceptionE(vim.fault.GenericVmConfigFault)

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0

Cause

This issue happens due to the virtual machine status change to invalid during shutdown operation.

Resolution

Currently, there is no resolution.

To work around this issue, remove the invalid virtual machine from inventory list and add it again.

Also decrease the number of virtual machines shutdown operations at same time to under 10.