VM performance issues due to FC command timeouts and link failures

Products

• VMware vSphere ESXi
• VMware vSphere ESX 6.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

VM performance is impacted and ESXi logs report FC command timeouts, possibly against paths via one vmhba. 

The var/run/log/vmkernel.log file shows errors similar to:

cpu6:2097158)lpfc: lpfc_handle_status:5637: 1:(0):3271: FCP cmd x28 failed <1/1> sid x010700, did x010b01, oxid x25e iotag x584 Time Out Returning Host Busy
cpu14:2098209)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x89 (0x45b9432645c8, 2097209) to dev "naa.################################" on path "vmhba#:C0:T1:L#" Failed:
cpu14:2098209)NMP: nmp_ThrottleLogForDevice:3875: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x43084904ba00 CmdSN 0x14e
cpu14:2098209)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...
cpu14:2098209)ScsiDeviceIO: 4154: Cmd(0x45b9432645c8) 0x89, cmdId.initiator=0x43084904ba00 CmdSN 0x14e from world 2097209 to dev "naa################################" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer.

or

YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu11:2103492)lpfc: lpfc_handle_status:4260: vmhba3 3271: FCP cmd x8a failed <4/0> sid x087100, did x026b81, oxid x59c iotag x8c2 Time Out Returning Host Busy
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu11:2098464)lpfc: lpfc_handle_status:4260: vmhba3 3271: FCP cmd x8a failed <4/0> sid x087100, did x026b81, oxid x5de iotag x904 Time Out Returning Host Busy
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2097277)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x8a (0x45ba26750e00, 2103160) to dev "naa.################################" on path "vmhba3:C0:T4:L0" Failed:
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2097277)NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430ec52c9700 CmdSN 0x800b0002
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu44:2097277)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.################################" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2097277)ScsiDeviceIO: 4633: Cmd(0x45ba26750e00) 0x8a, CmdSN 0x800b0002 from world 2103160 to dev "naa.################################" failed H:0x2 D:0x0 P:0x0

FC stats report high Link Failure Count for one or more HBAs

# esxcli storage san fc stats get 

FcStat:
   Adapter: vmhba#
   Tx Frames: 19687
   Rx Frames: 693458
   Lip Count: 0
   Error Frames: 0
   Dumped Frames: 0
   Link Failure Count: 793
   Loss of Signal Count: 0
   PrimSeq Protocol Err Count: 0
   Invalid Tx Word Count: 186
   Invalid CRC Count: 0
   Input Requests: 0
   Output Requests: 0
   Control Requests: 0

Invalid CRC count may also be high.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This pattern of error indicates a likely FC cabling, SFP or interconnect issue.

Resolution

Investigate on Fibre Channel fabric.