The host is losing storage access because of firmware resets. The vmkernel shows the error message: "iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED."

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms: 

• Host in a cluster lost access to all the datastores
• There were numerous entries in VMkernel logs with similar messages:

cpu0:2098181)nfnic: <1>: INFO: fnic_handle_link: 943: link status 0 down cnt 1
cpu0:2098181)nfnic: <1>: INFO: fnic_handle_link: 946: old status 1 old down cnt 0
cpu0:2098181)nfnic: <1>: INFO: fnic_handle_link: 1017: fnic1: recvd up to down event
cpu0:2098181)nfnic: <1>: INFO: fnic_fdls_link_status_change: 99: fnic1: FDLS link status change link up:0, usefip:0
cpu0:2098181)nfnic: <1>: INFO: fnic_fdls_link_down: 4504: Moving tport to PLOGO state tport: 1 fcid: 0x10400
cpu0:2098181)nfnic: <1>: INFO: fnic_fdls_link_down: 4504: Moving tport to PLOGO state tport: 2 fcid: 0x11400
cpu0:2098181)WARNING: nfnic: <1>: fnic_fw_reset_handler: 307: Issued fw reset


### Issued fw reset indicates the firmware crash 

2024-08-22T18:53:04.289Z In(182) vmkernel: cpu47:4997414)nfnic: <2>: INFO: fnic_taskMgmt: 2080: VMK_SCSI_TASKMGMT_ABORT issued. iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED
2024-08-22T18:53:04.289Z In(182) vmkernel: cpu47:4997414)nfnic: <1>: INFO: fnic_taskMgmt: 2080: VMK_SCSI_TASKMGMT_ABORT issued. iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED

### At the same time logs shows FNIC_IPORT_IO_BLOCKED

• Number of Events that are logged for FNIC_IPORT_IO_BLOCKED in the vmkernel logs
  
  # less vmkernel.*** | grep -i FNIC_IPORT_IO_BLOCKED | wc -l
314115

Environment

vSphere ESXi 6.x
vSphere ESXi 7.x
vSphere ESXi 8.x

Cause

• The logs shows firmware crash. Indicates the issue with the Firmware 
• "state: 3" means FNIC_IPORT_STATE_FABRIC_DISC (fabric discovery stage).  
• FNIC_IPORT_IO_BLOCKED messages means iport is not in ready state. Such log message could be triggered by various factors , i.e. during Fabric Discovery stage(state 3) , there was a burst of IOs sending to the iport which is not ready, etc. 

Resolution

 

• Refer this KB as general guideline when decide I/O device firmware and driver : "Supported drivers and firmware version for I/O devices"
• Engage hardware vendor to analyze firmware dump to further investigation and recommendations.