vMotion fails at 10% with the error: Operation timed out

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• When you attempt to use vMotion to migrate a virtual machine, the task fails at 10%.
  
  
• In virtual machine's vmware.log file, you see these errors:
  
  vmx| MigrateStatusFailure: Now in new log file.
  vmx| MigrateStatusFailure: Timed out waiting for migration start request.
  
  
• In the source ESXi/ESX host's hostd.log file, you see errors similar to:
  
  vm:/vmfs/volumes/<Datastore>/<VM>.vmx' 128215984 warning] Failed to find activation record, event user unknown.
  
  Message on <VM> on <HOST> in ha-datacenter: Cannot open/create log file '/vmfs/volumes/<Datastore>/<VM>/vmware.log': Device or resource busy
  
  Unable to open log file "/vmfs/volumes/<Datastore>/<VM>/vmware.log". Check your configuration to make sure that the path specified for the log file is valid, and that you have write privileges in this directory.
  
  
• In the destination ESXi/ESX host's VMkernel file, you see these errors multiple times:
  
  Checking if lock holders are live for lock [type 10c00001 offset 22906880 v 235, hb offset 4097536
  
  Exceeded retries removing vmware-0.log from f530 28 2 47c7a4e6 d89cf356 1400ffb9 56471c5e 3c0c244 14 0 0 0 0 0
  
  
• When you examine the virtual machine's .vmx file, you see that the vmware.log rotation size is set to a very low value. For example:
  
  log.rotateSize = "500"

Environment

VMware vCenter Server 4.0.x
VMware vCenter Server 5.0.x
VMware VirtualCenter 2.5.x
VMware vCenter Server 4.1.x

Cause

By default, the virtual machine log file (vmware.log) is rotated as a result of the virtual machine's Power On or Power Off operation. To configure log rotation based on file size, the log.rotateSize line is included in the virtual machine's .vmx file. For example:

log.rotateSize = <maximum size in bytes the file can grow to>

When the log.rotateSize value in the virtual machine's .vmx file is set to a very low value, it causes the vmware.log file to rotate so quickly that by the time the destination host is requesting the vmware.log file's VMFS lock, it has already rotated through and there is a new vmware.log file associated with a new inode in the file system. The destination host is then unable to acquire a proper file lock, and this causes the vMotion migration failure.

Resolution

To resolve this issue, modify the log.rotateSize value to a larger number to prevent the log file from rotating too quickly:

1. Log into the ESXi/ESX host or the vCenter Server using the vSphere Client.
2. Power off the virtual machine.
3. Click Edit Settings on the virtual machine's Summary page, then click the Options tab.
4. Under Advanced, click General.
5. Click Configuration Parameters > Configuration.
6. Find the log.rotateSize parameter and change the setting to a larger value.
   
   Note: Do not change the log.keepOld setting to a large number or the datastore may fill up. The default value for log.keepOld is 6.
   
   
7. Click OK.
8. Power on the virtual machine.
9. Verify that the issue is resolved by using vMotion to migrate the virtual machine.

Additional Information

For related information, see Log rotation and logging options for vmware.log (8182749).

To troubleshoot similar issues, see Diagnosing a vMotion failure at 10% in vCenter Server (1003734).
Understanding and troubleshooting vMotion
Log rotation and logging options for vmware.log

Impact/Risks:
Be cautious about making major changes to your vmware.log settings, as they could easily cause your datastores to fill up.