Failed to resume IO’s after continuous target port block operation

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• If a server running ESXi 7.0 and has Emulex FC HBAs installed with NVMe support enabled in the adapter, IO’s might not resume after continuous toggling of the Initiator or Target port
• In the /var/run/log/vmkernel.log file, you see entries similar to:

2020-02-27T04:44:06.458Z cpu5:1048581)WARNING: NVMEPSA:63 Complete vmkNvmeCmd: <pointer>, vmkPsaCmd: <pointer>, 
cmdId.initiator=<pointer>, CmdSN: <command sqe num>, status: 0x80a
2020-02-27T04:44:06.458Z cpu12:1049686)WARNING: error H:0x3

Environment

VMware vSphere ESXi 7.0.0

Cause

When an IO does not complete from the target, the brcmnvmefc/lpfc driver returns IO to PSA-NVMe layer with the status “VMK_NVME_STATUS_VMW_TIMEOUT”. PSA NVMe layer does not retry IO completed with “VMK_NVME_STATUS_VMW_TIMEOUT”.
Reasons of IO not completing from target end

• IO timeout
• IO Aborted by NVMe-FC Transport/LLD driver
• Port Toggle
• Initiator Port Link Down.

Resolution

To resolve this issue, upgrade brcmnvmefc/lpfc driver to a version higher than 12.6.228.4.