Determining if a High Availability Virtual Machine Monitoring event caused a virtual machine to reboot

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article provides information about the logs and events to determine if the reboot was caused by HA Virtual Machine Monitoring.

Symptoms:

• Virtual machine running on an ESX/ESXi host is rebooted unexpectedly
• The ESX/ESXi host where the virtual machine is registered is a part of an VMware High Availability (HA) cluster in vCenter Server and Virtual Machine Monitoring is enabled on the HA cluster

Resolution

To look for events related to the virtual machine in vCenter Server:

1. Select the virtual machine in vCenter Server.
2. Click the Tasks & Events tab.
3. Click the Events button.
4. Enter the word reset in the search field.
5. If HA Virtual Machine Monitoring was responsible for resetting the virtual machine, you see an Event similar to:
   
   This virtual machine reset by HA. Reason: VMware Tools heartbeat failure. A screenshot is saved at /vmfs/volumes/########-######10-28d9-00215-a5d36b8/vm/vm-screenshot-0.png
   
   Note: The reason in the Description field may differ slightly, but it always states that the virtual machine was reset by HA if Virtual Machine Monitoring is involved.

On the ESX/ESXi host, two logs can help identify Virtual Machine Monitoring as the source of the virtual machine reboot. If Virtual Machine Monitoring is the cause:

• In the /var/log/vmware/hostd.log file, you see entries similar to:
  
  T F5760B90 verbose 'vm:/vmfs/volumes/########-####-########36b8/vm/vm.vmx'] Updating current heartbeatStatus: red
  T F5B73B90 verbose 'vm:/vmfs/volumes/########-####-########36b8/vm/vm.vmx' opID=task-internal-977-4429b01] Reset request received
• In the /vmfs/volumes///vmware.log file of the affected virtual machine, you see entries similar to:
  
  T: vmx| Vix: [104333 vmxCommands.c:392]: VMAutomation_Reset
  T: vmx| Vix: [104333 vmxCommands.c:457]: VMAutomation_Reset. Trying hard reset
  T: vmx|
  T: vmx|
  T: vmx| VMXRequestReset
  T: vmx| Stopping VCPU threads...

Note: Knowing the time of the unexpected reboot helps in searching the log files, as you can look for the time stamp.

If any of the relevant log entries or the vCenter Server event are present, HA Virtual Machine Monitoring restarted the virtual machine as it was not receiving virtual machine heartbeats (via VMware Tools) and there was no I/O on the virtual machine.

Examine the logs within the guest operating system to help determine the cause of the event. If multiple virtual machines are affected, review the vmkernel, messages, and hostd logs on the ESX host where the virtual machines are registered to look for a system wide problem.

To workaround this issue, disable Virtual Machine Monitoring:

• Log in to the vCenter Server using the vSphere Client.
• Right-click on VMware HA Cluster and select Edit Settings
• In the Cluster Settings dialog box, Select VM Monitoring
• VM Monitoring Status: VM Monitoring drop down box, select Disabled
• Click OK

Note: This feature can also be disabled on the VMware HA page of the New Cluster Wizard by deselecting Enable Host Monitoring.