VM is randomly in a powered off state - vCLS VM Lifecycle unexpectedly powered off a non-vCLS VM.

Products

• VMware vSphere ESXi

Issue/Introduction

• VMs are randomly found in a powered off state without any user-initiated operation.
• A review of the /var/run/log/hostd.log confirms that the virtual machine was powered off and the associated opID indicates that the power-off operation was initiated as part of the vCLS workflow.
  
  YYYY-MM-DDTHH:MM:SS info hostd[2100025] [Originator@6876 sub=Vimsvc.TaskManager opID=wcp-vCLS-##-##-#### user=vpxuser:VSPHERE.LOCAL\vpxd-extension-#######-####-####-####-##########] Task Created : haTask-7-vim.VirtualMachine.powerOff-#######
YYYY-MM-DDTHH:MM:SS verbose hostd[2100026] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-#######-####-########/vmname/vmname.vmx opID=wcp-vCLS-##-##-#### user=vpxuser:VSPHERE.LOCAL\vpxd-extension-#######-####-####-####-##########] Power off request received

Cause

This issue arose from a known issue in the vpxa service restart workflow as explained in KB VM suddenly appears to be in an unexpected power state after vpxa service restart on the host  which coincided with the automated password rotation task for vCLS virtual machines.

Due to this known issue, a non-vCLS virtual machine was incorrectly mapped to vCLS VM ID resulting in an unintended power-off operation that was meant for a vCLS VM.


Reviewing the logs confirms vpxa service restart and vCLS VM lifecycle operation at the same time.

• A host level certificate refresh was triggered which initiated VPXA service restart (/var/run/log/hostd.log on the host) :
  YYYY-MM-DDTHH:MM:SS info hostd[3152719] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=WorkQueue-######-#### user=vpxuser] Event 9958 : User vpxuser@##.##.##.## has successfully replaced the CA and the revocation list.
YYYY-MM-DDTHH:MM:SS- time the service was last started YYYY-MM-DDTHH:MM:SS, Section for VMware ESXi, pid=#####, version=7.0.3, build=21930508, option=Release


• During the same timeframe, the vCLS password expired in the background, triggering an automated password rotation task that initiated the recreation of the vCLS virtual machines (/var/log/vmware/wcp.log on vCenter):.
  YYYY-MM-DDTHH:MM:SS info wcp [password/###:120] [opID=vCLS] Password is already expired, rotating the password
YYYY-MM-DDTHH:MM:SS info wcp [clustersvc/#####:85] [opID=vCLS] Password is rotated. Time to change the password

Resolution

This is known issue with vSphere 7.x and 8.x. For more information, please refer : KB VM suddenly appears to be in an unexpected power state after vpxa service restart on the host

Upgrading ESXi to ESXi 8.0 U2 and above will prevent the vpxa service restart during Host-level certificate refresh.