VM performance issues due to FC command timeouts and link failures

Products

• VMware vSphere ESXi
• VMware vSphere ESX 6.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

VM performance is impacted and ESXi logs report FC command timeouts, possibly against paths via one vmhba. 

vmkernel.log shows errors similar to:

cpu6:2097158)lpfc: lpfc_handle_status:5637: 1:(0):3271: FCP cmd x28 failed <1/1> sid x010700, did x010b01, oxid x25e iotag x584 Time Out Returning Host Busy
cpu14:2098209)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x89 (0x45b9432645c8, 2097209) to dev "naa.################################" on path "vmhba#:C0:T1:L#" Failed:
cpu14:2098209)NMP: nmp_ThrottleLogForDevice:3875: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x43084904ba00 CmdSN 0x14e
cpu14:2098209)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...
cpu14:2098209)ScsiDeviceIO: 4154: Cmd(0x45b9432645c8) 0x89, cmdId.initiator=0x43084904ba00 CmdSN 0x14e from world 2097209 to dev "naa################################" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer.

FC stats report high Link Failure Count for one or more HBAs

# esxcli storage san fc stats get 

FcStat:
   Adapter: vmhba#
   Tx Frames: 19687
   Rx Frames: 693458
   Lip Count: 0
   Error Frames: 0
   Dumped Frames: 0
   Link Failure Count: 793
   Loss of Signal Count: 0
   PrimSeq Protocol Err Count: 0
   Invalid Tx Word Count: 186
   Invalid CRC Count: 0
   Input Requests: 0
   Output Requests: 0
   Control Requests: 0

 

Invalid CRC count may also be high.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This pattern of error indicates a likely FC cabling, SFP or interconnect issue.

Resolution

Investigate on Fibre Channel fabric.