vMotion fails at 82% with the hostd log error: "Source detected that destination failed to resume"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vMotion fails at 82%
• Cannot migrate a virtual machine with vMotion
• In the /var/log/vmware/hostd.log file on source ESX host, you see the error:
  
  ResolveCb: Failed with fault: (vmodl.fault.SystemError) {
  reason = "Source detected that destination failed to resume."
  msg = ""
  }
   
• In the /var/log/vmware/hostd.log file, you see the entry:
  
  Upgrade is required since hwVersion in config file is 3

Environment

VMware ESX Server 3.5.x
VMware ESXi 4.1.x Installable
VMware ESX 4.1.x
VMware ESXi 3.5.x Embedded
VMware ESXi 4.1.x Embedded
VMware ESX 4.0.x
VMware ESXi 4.0.x Embedded
VMware ESXi 3.5.x Installable
VMware ESX Server 3.0.x
VMware ESXi 4.0.x Installable

Cause

This issue occurs due to incorrect datastore information or out-of-date virtual hardware.

 

Note: A similar issue is seen when there is a misconfiguration of the swap file locations between the ESXi/ESX host and vCenter Server. For more information on configuring swap files, see Storing a virtual machine swap file in a location other than the default in ESX/ESXi (1004082).

Resolution

Incorrect datastore information

To determine if this issue is caused by incorrect datastore information, run this command on your ESX hosts:

# vdf -h

You see output similar to:

 

[root@esx01 ~]# vdf -h
/vmfs/volumes/1dd794c6-cc279de7
600G 438G 161G 73% /vmfs/volumes/VM_STOR2
/vmfs/volumes/e2978cef-5a532852
400G 282G 117G 70% /vmfs/volumes/VM_STOR7

[root@esx02 ~]# vdf -h
/vmfs/volumes/1dd794c6-cc279de7
600G 438G 161G 73% /vmfs/volumes/VM_STOR2
/vmfs/volumes/e2978cef-5a532852
400G 282G 117G 70% /vmfs/volumes/VM_STOR7

[root@esx03 ~]# vdf -h
/vmfs/volumes/36132c1c-6f72083e
600G 437G 162G 72% /vmfs/volumes/VM_STOR2
/vmfs/volumes/e01dc51c-0e591900
400G 282G 117G 70% /vmfs/volumes/VM_STOR7

 

In this example, the third host identifies the datastore with a different UUID. Adding a host by these four different methods produces four different UUIDs:

• hostname
• hostname.domainname
• IP.address.of.host
• HOSTNAME vs. hostname

Run this command to find which methods were used to mount the datastores, SSH into each host:

# esxcfg-nas -l

To resolve this issue:

• If the virtual machine(s) that are attached to the datastore(s) in question can be powered off:
   
  1. Power off the virtual machine(s).
  2. Unmount the NFS datastore.
  3. Mount the NFS datastore with the required details to be consistent with all the hosts that have access to this datastore (that is, HOSTNAME, hostname, hostname.domainname, or IP.address.of.host).
  4. Power on the virtual machine(s).
      
• If the virtual machine(s) cannot be powered off:
   
  1. Use Storage vMotion to move virtual machines to a volume that is equally shared by the hosts. For more information, see Moving virtual machines with Storage vMotion (1005544).
  2. Unmount the NFS datastore.
  3. Mount the NFS datastore with the required details to be consistent with all the hosts that have access to this datastore (that is, HOSTNAME, hostname, hostname.domainname, or IP.address.of.host).
  4. Move the virtual machine(s) back to the datastore(s) using Storage vMotion.

Note: This procedure is applicable for NFS datastores, with block storage (local, Fiber Channel, iSCSI) steps 2 and 3 can be replaced with unpresenting/re-presenting the LUN from the ESX/ESXi host. For more information, see Removing a LUN containing a datastore from VMware ESXi/ESX 4.x (1029786)

If the issue persists:

1. Unmap and remap the LUN from the storage side.
2. Confirm that the UUID displays correctly, both from an SSH Client and in the datastore of the host.
3. In the var/log/vmkwarning file, look for an entry similar to:
   
   Failed to migrate vswp file" with /vmfs/ incorrect_datastore_ID
    
4. If you see this entry, power off the virtual machine.
5. Right-click the virtual machine in the Inventory and choose Edit Settings.
6. Click the Option tab.
7. Select a swap file location.
8. Select Always store with the virtual machine.

Out-of-date virtual hardware

To determine if this issue is caused by out-of-date hardware, in the /var/log/vmware/hostd.log file look for an entry similar to:

Upgrade is required since hwVersion in config file is 3

This log entry indicates that the virtual machine's virtual hardware is at an earlier version than is required.

To confirm that the virtual hardware is out-of-date:

1. Right-click on the virtual machine in the Inventory and choose Edit Settings.
2. Check in the upper right corner of the Virtual Machine Properties screen. If the version is less than 4, you must upgrade the virtual hardware for this virtual machine.

Additional Information

Storing a virtual machine swap file in a location other than the default in ESX/ESXi
Moving virtual machines with Storage vMotion
Removing a LUN containing a datastore from VMware ESXi/ESX 4.0 and 4.1
vMotion が 82% で次の hostd ログ エラーで失敗する： Source detected that destination failed to resume.
vMotion 在完成 82% 时失败并显示 hostd 日志错误：源检测到目标无法恢复。