Unable to vMotion VM following ESXi upgrade

Products

• VMware vSphere ESXi

Issue/Introduction

• vMotion task failed for VMs on host with message "Failed to establish transport connection"
  
  
• /var/log/vmkernel.log reports logging similar to:
  
  lpfc: lpfc_handle_status:5637: 2:(0):3271: FCP cmd xa3 failed <1/3> sid x9d0100, did x9d0300, oxid xc3b iotag xf61 Abort Requested Host Abort Req
NMP: nmp_ThrottleLogForDevice:3867: Cmd 0xa3 (0x45d9219f4f48, 0) to dev "naa.####" on path "vmhba#:C#:T#:L#" Failed:
NMP: nmp_ThrottleLogForDevice:3875: H:0x5 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x4538d839bb68 CmdSN 0x0
VMW_SATP_ALUA: satp_alua_issueCommandOnPath:873: Path "vmhba#:C#:T#:L#" (UP) command 0xa3 failed with status Timeout. H:0x5 D:0x0 P:0x0 .
WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.####" state in doubt; requested fast path state update...
ScsiDeviceIO: 4115: Cmd(0x45b90199d408) 0x2a, CmdSN 0x4d from world 2099410 to dev "naa.####" failed H:0x2 D:0x0 P:0x0
lpfc: lpfc_handle_status:5637: 2:(0):3271: FCP cmd x2a failed <1/2> sid x9d0100, did x9d0300, oxid x28a iotag x5b0  Returning Host Busy
  
  
  
• FC stats report increasing Invalid Tx Word Count for HBAs

         #esxcli storage san fc stats get 

FcStat:
   Adapter: vmhba#
   Tx Frames: 222041
   Rx Frames: 2313873
   Lip Count: 0
   Error Frames: 0
   Dumped Frames: 0
   Link Failure Count: 0
   Loss of Signal Count: 0
   PrimSeq Protocol Err Count: 0
   Invalid Tx Word Count: 99
   Invalid CRC Count: 0
   Input Requests: 0
   Output Requests: 0
   Control Requests: 0

Environment

VMware ESXi 7.0.x

Resolution

• Ensure that the FC HBA driver and firmware are supported as per the Broadcom HCL. Note that on upgrade a newer driver may be installed which is not supported with the existing firmware.
• Investigate on the fabric for the cause of observed aborts/resets, increasing TX Word Count. 

Additional Information

For H:0x2 and H:0x5 refer to article Understanding SCSI host-side NMP errors/conditions in ESXi

For "state in doubt" refer to article "state in doubt; requested fast path state update" error in ESXi