vMotion fails at 19% with error 'Operation Timed out'

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machine vMotion fails at 19% intermittent and the task error is 'Operation Timed out'
• The virtual machine /vmfs/volumes/datastore-uuid/vm-name/vmware.log:
  
  YYY-MM-DDTHH:MM:SS.720Z| vmx| | I005: [msg.migrate.expired] Timed out waiting for migration start request


• The destination host /var/run/log/hostd.log:
  
  YYY-MM-DDTHH:MM:SS.930Z info hostd[2100822] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<datastore-uuid>/<vm-name>/<vm-name>.vmx opID=<opid> user=vpxuser:<user-name>] GetWid: timed out
YYY-MM-DDTHH:MM:SS.936Z error hostd[2100822] [Originator@6876 sub=Vcsvc.VMotionDst.6817728631801283937 opID=<opid> user=vpxuser:<user-name>] Initiate: Failed to get the WID
  
  
• The destination host /var/run/log/vpxa.log:
  
  YYY-MM-DDTHH:MM:SS.254Z info vpxa[2100769] [Originator@6876 sub=Default opID=<opid>] [VpxLRO] -- ERROR lro-4031961 -- vmotionManager -- vim.host.VMotionManager.initiateDestination: vim.fault.Timedout:
--> Result:
--> (vim.fault.Timedout) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<cs p:000000ff2f6fd610, TCP:localhost:8307>]: initiateDestination
--> Operation timed out."
  
  
• The storage in destination host has high IO latency in /var/run/log/vmkernel.log:
  
  YYY-MM-DDTHH:MM:SS.865Z cpu72:2098280)WARNING: ScsiDeviceIO: 1596: Device <LUN-ID> performance has deteriorated. I/O latency increased from average value of 3639 microseconds to 80409 microseconds.
  
  
• The storage device of datastore where the virtual machine stays is larger than 64TB.

Environment

• VMware vSphere 7.0.x
• VMware vSphere 8.0.x

Cause

The large datastore size and high storage device latency cause the IO commands of vMotion using more than 90s. The time 90s is that vMotion can wait for the destination virtual machine starting up.  

Resolution

• Recreate datastore with smaller size. The VMFS6 supports the maximum size 64TB. See vSphere 7.0 Configuration Limits
• Engage storage vendor to check the storage performance to reduce the latency. See Troubleshooting poor Storage Controller performance for VMware ESXi

Additional Information

Using esxtop to identify storage performance issues for ESXi (multiple versions)