Failed to relocate virtual machine from one host to another host with error: Operation timed out.

Products

• VMware vSphere ESXi

Issue/Introduction

• It failed to relocate virtual machine from one host to another host with error: Operation timed out.
• From the var/run/log/vmkernel.log, the following events may be seen:
  YYYY-MM-DDTHH:MM:SS] cpu52:2100### opID=######)VmMemXfer: vm ######: 2462: Could not find MemXferFS region for /vmfs/volumes/######68e8/####/####.vmx
  [YYYY-MM-DDTHH:MM:SS] cpu48：2100### opID=######）Fil3:5033:Lock failed on file:###.vmx on vol '<datastore-name>' with FD:<FD c42 r4>
  
  [YYYY-MM-DDTHH:MM:SS] cpu40：######) qlnativefc:vmhba# (12:2.0): qlnativefcStatusEntry:1924:(0:0) Dropped frame(s) detected (118784 of 131072 bytes).
  [YYYY-MM-DDTHH:MM:SS] cpu40：######) qlnativefc:vmhba# (12:2.0): qlnativefcStatusEntry:2079:C#:T#:L# - FCP command status:0x15-0x0 (0x2) portid=xxxxxx oxid=xxxxxcdb=###### len=##### rspInfo=0x0 resid=0x0 fwResid=0x1d000 host status = 0x2 device sta$
  [YYYY-MM-DDTHH:MM:SS] cpu40：######) qlnativefc:vmhba# (12:2.0): qlnativefcStatusEntry:1924:(0:0) Dropped frame(s) detected (73728 of 131072 bytes).
  [YYYY-MM-DDTHH:MM:SS] cpu40：######) qlnativefc:vmhba# (12:2.0): qlnativefcStatusEntry:2079:C#:T#:L# - FCP command status:0x15-0x0 (0x2) portid=xxxxxx oxid=xxxxxcdb=###### len=##### rspInfo=0x0 resid=0x0 fwResid=0x12000 host status = 0x2 device sta$
  [YYYY-MM-DDTHH:MM:SS] cpu41：######) ScsiDeviceIO: 4115 :Cmd (0x45d901a6b4c8) 0x28, CmdSN oxa from world 2099990 dev "naa.###############000000" failed H;0x2 D:0x0 P:0x0...
  [YYYY-MM-DDTHH:MM:SS] cpu41：######) WARNING: NMP :NMP device "naa.###############000000" state in doubt; requested fast path state update...
  
  
  

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

VM is running on datastore:<datastore-name>, and the underlying device is naa.###############000000
Host Status [H:0x2] :This status is returned when the HBA driver is unable to issue a command to the device naa.###############000000. This status occurs due to dropped FCP frames in the environment.

Resolution

To engage HBA hardware vendor to investigate and mitigate the issue.
For workaround, to remove the FC cable from vmhba2 and see whether issue can be resolved.

Additional Information

For [H:0x2] refer article: Understanding SCSI host-side NMP errors/conditions in ESXi