Virtual machines may unexpectedly reboot when using VMware HA with virtual machine monitoring on ESX 3.5 Update 3
Article ID: 308126
Updated On:
Products
VMware vCenter Server
VMware vSphere ESXi
Issue/Introduction
Symptoms:
- Virtual machines may unexpectedly reboot after a VMotion migration to an ESX 3.5 Update 3 (build 123630) host.
- Virtual machines may unexpectedly reboot after a Power On operation on an ESX 3.5 Update 3 (build 123630) host, when the VMware High Availability (HA) feature with virtual machine monitoring is active.
- This issue may following the periodic vpxuser password update and service restart, as shown in the logs:
[2009-06-03 11:18:55.759 'TaskManager' 110087088 info] Task Created : haTask-ha-folder-root-vim.host.LocalAccountManager.updateUser-1870238977
[2009-06-03 11:19:18.132 'App' 3076440992 info] END SERVICES ....
[2009-06-03 11:20:45.968 'App' 65657776 verbose] [vpxaVmFailover::NotifyVmFailover] VM 2, time 1244042445, failure type 0, interval 40, uptime 4410909
[2009-06-03 11:20:45.968 'App' 65657776 verbose] [vpxaVmFailover::NotifyVmFailover] VM 6, time 1244042445, failure type 0, interval 40, uptime 6678154
[2009-06-03 11:20:45.968 'App' 65657776 verbose] [vpxaVmFailover::NotifyVmFailover] VM 7, time 1244042445, failure type 0, interval 40, uptime 5941025
[2009-06-03 11:20:45.968 'App' 65657776 verbose] [vpxaVmFailover::NotifyVmFailover] VM 8, time 1244042445, failure type 0, interval 40, uptime 66781 54
[2009-06-03 11:20:45.968 'App' 65657776 verbose] [vpxaVmFailover::NotifyVmFailover] VM 9, time 1244042445, failure type 0, interval 40, uptime 4737749
- /var/log/vmware/hostd.log shows:
[2009-03-20 04:44:35.252 'TaskManager' 3076453280 info] Task Created : haTask-512-vim.VirtualMachine.reset-47992
[2009-03-20 04:44:35.323 'ha-eventmgr' 3076453280 info] Event 8420 : Win2003-1 on esx1.xyz.com in ha-datacenter is reset
[2009-03-20 04:44:35.323 'vm:/vmfs/volumes/48331160-05c64c5c-edf0-001e0bd8c708/Win2003-1/Win2003-1.vmx' 3076453280 info] State Transition (VM_STATE_ON -> VM_STATE_RESETTING)
Environment
VMware ESX Server 3.5.x
VMware VirtualCenter 2.5.x
VMware VirtualCenter 2.5.x
Resolution
A virtual machine may reboot itself if:
- The virtual machine is running on a ESX 3.5 Update 3 host, either by virtue of VMotion or a Power On operation
AND
- The host has VMware HA enabled with the virtual machine monitoring option active.
Virtual machine monitoring is dependent on VMware tools heartbeats to determine the state of the virtual machines.
With ESX Server 3.5 Update 3 after a VMotion or a Power On operation, the host agent running on the ESX server may delay sending the heartbeat state of the virtual machine to the Host. VMware HA detects this as a failure of the virtual machine and attempts to restart the virtual machine.
The issue has been resolved in VirtualCenter 2.5 Update 4. You can download VirtualCenter 2.5 Update 4 from the Download Center.
If you cannot upgrade, perform one of the following workarounds.
Workaround 1: Disabling virtual machine monitoring
To disable virtual machine monitoring:
- Right-click the VMware HA cluster and choose Edit Settings.
Note: This feature can also be enabled for a new cluster on the VMware HA page of the New Cluster wizard. - In the Cluster Settings dialog box, select VMware HA in the left column.
- Deselect Enable virtual machine monitoring.
- Click OK.
Workaround 2: Setting hostd heartbeat delay to 0
Notes:
- If you perform workaround 2, the heartbeat alarm in VirtualCenter (for virtual machines running on the ESX 3.5 Update 3 host) may go red until the VMware Tools inside GOS starts to heartbeat.
- Before you begin please refer to Restarting the Management agents on an ESX Server (1003490) for important information on restarting the mgmt-vmware service.
To set the hostd heartbeat delay to 0:
- Disconnect the host from VirtualCenter. Right-click on the host in VMware Infrastructure (VI) Client and choose Disconnect.
- Log in as root to the ESX host with an SSH client.
- Open the file /etc/vmware/hostd/config.xml in a text editor.
- Set the heartbeatDelayInSecs tag under vmsvc to 0 seconds:
<vmsvc>
<heartbeatDelayInSecs>0</heartbeatDelayInSecs>
<enabled>true</enabled>
</vmsvc> - Save your changes and exit.
- Restart the management agents for this change to take effect. For more information, see Restarting the Management agents on an ESX Server (1003490).
- Reconnect the host in VirtualCenter. Right-click on the host in VI Client and choose Connect.
Notes:
- Setting heartbeatDelayInSecs to 0 does not disable virtual machine monitoring.
- When any ESX patches are applied, changes to the config.xml file are over-written.
Additional Information
Feedback
Yes
No
Powered by
