"An error occurred while communicating with the remote host", vCLS VM will not Power Off and environment is getting flooded with errors when vCenter is trying to power down a vCLS VM

Products

• VMware vCenter Server 8.0

Issue/Introduction

• vCenter is trying to Power off the vCLS VMs continuously and VC events are flooded with the error message "An error occurred while communicating with the remote host" during the Power Off operation.
• Recent Tasks shows failed vCLS Power Off tasks every few seconds as in below image : 
  
  
  
  
• Unable to manually Power Off the vCLS VMs.
• HA and DRS are disabled in the Cluster where the specific vCLS VMs are located.
• There are no disconnected hosts in the Cluster.
• VPXD log shows similar to below error message during the task failure:
  
  YYYY-MM-DDTHH:MM:SS.000Z error vpxd[pid] [Originator@6876 sub=Vmomi opID=<opid>] Cannot contact the specified host (<hostname>)
YYYY-MM-DDTHH:MM:SS.000Z error vpxd[pid] [Originator@6876 sub-vmmoVm opID-<opid>] Powering off VM [vim.VirtualMachine:vm-<MoID>, vCLS-<ID>] failed with exception N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication)

Cause

This issue is caused when the ESXi hosts where vCLS VMs are running is Out-of-Sync with the vCenter. ESXi host summary tab in vSphere Client will show the message "Cannot synchronize host <hostname>" if it is out of sync.

Resolution

Disconnect and reconnect the host by following the instructions in Host Summary page reports, Cannot synchronize host or Quick stats on <Hostname> is not up-to-date.