A specific virtual machine was restarted by vSphere HA.

Products

• VMware vCenter Server
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

• A specific virtual machine was restarted by vSphere HA. No problems were observed on the ESXi host on which the virtual machine was running, and other virtual machines on the same host are running normally.
• "VM Monitoring" is enabled in vSphere HA on the cluster that contains the virtual machines.
• The following message was output to the fdm.log on the ESXi host.

YYYY-MM-DDTHH:MM:SS.sssZ verbose fdm[####] [Originator@#### sub=Policy opID=########] VM /vmfs/volumes/##################################/VM_NAME/VM_NAME.vmx: guest heartbeat transitioned to red
YYYY-MM-DDTHH:MM:SS.sssZ verbose fdm[####] [Originator@#### sub=Policy] IO metrics value is 0 at # for metric ######
YYYY-MM-DDTHH:MM:SS.sssZ verbose fdm[####] [Originator@#### sub=Policy] Reset vm /vmfs/volumes/##################################/VM_NAME/VM_NAME.vmx: true
YYYY-MM-DDTHH:MM:SS.sssZ verbose fdm[####] [Originator@#### sub=Invt] GuestKernelCrashed is false for VM ##
YYYY-MM-DDTHH:MM:SS.sssZ info fdm[####] [Originator@#### sub=Policy opID=################] VM /vmfs/volumes/#######################################/VM_NAME/VM_NAME.vmx has been reset

• The following message was output to the vmware.log file of the virtual machine restarted by vSphere HA.

YYYY-MM-DDTHH:MM:SS.sssZ| vmx| I005: GuestRpcSendTimedOut: message to toolbox timed out.
YYYY-MM-DDTHH:MM:SS.sssZ| vcpu-0| I005: Tools: Tools heartbeat timeout.
YYYY-MM-DDTHH:MM:SS.sssZ| vmx| I005: GuestRpcSendTimedOut: message to toolbox timed out.
YYYY-MM-DDTHH:MM:SS.sssZ| vmx| I005: GuestRpc: app toolbox's second ping timeout; assuming app is down
YYYY-MM-DDTHH:MM:SS.sssZ| vmx| I005: Tools: [AppStatus] Last heartbeat value ######## (last received ##s ago)
YYYY-MM-DDTHH:MM:SS.sssZ| vmx| I005: TOOLS: appName=toolbox, oldStatus=#, status=#, guestInitiated=#.
YYYY-MM-DDTHH:MM:SS.sssZ| vmx| I005: GuestRpc: Reinitializing Channel #(toolbox)
YYYY-MM-DDTHH:MM:SS.sssZ| vmx| I005: GuestMsg: Channel #, Cannot unpost because the previous post is already completed 
YYYY-MM-DDTHH:MM:SS.sssZ vmx - VigorTransportProcessClientPayload: opID=vmOperationsManager.cpp:###-vimOperationQueue.h:### seq=###: Receiving PowerState.InitiateReset request.
YYYY-MM-DDTHH:MM:SS.sssZ vmx vmOperationsManager.cpp:###-vimOperationQueue.h:388-48ec0aee-7820 Vix: [vmxCommands.c:##]: VMAutomation_Reset. Trying hard reset

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

An issue may have occurred on the guest OS side, causing it to crash.

When "VM Monitoring" is enabled in vSphere HA, vSphere HA monitors heartbeats from the VMware Tools process running on the guest OS. If a VMware Tools heartbeat is not received within the configured time, the virtual machine is restarted individually.

For more information on monitoring virtual machines with vSphere HA, please refer to the following documentation:

VM and Application Monitoring

Resolution

To resolve this issue, please conduct a thorough investigation to determine whether any issues existed on the guest OS side. In particular, reviewing the guest OS logs and performance data around the time of the reboot and thoroughly analyzing the state of the guest OS at the time of HA will help identify the root cause.