Host loses storage access with firmware resets and "iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED."

Products

• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

Symptoms: 

• One host in a cluster loses access to all of the datastores.
• Host went to hung state, anable to run any command
• VMs on host went to unreachable. 
• There are numerous entries in VMkernel logs with messages similar to the following:

cpu0:XXXXXXX)nfnic: <1>: INFO: fnic_handle_link: 943: link status 0 down cnt 1
cpu0:XXXXXXX)nfnic: <1>: INFO: fnic_handle_link: 946: old status 1 old down cnt 0
cpu0:XXXXXXX)nfnic: <1>: INFO: fnic_handle_link: 1017: fnic1: recvd up to down event
cpu0:XXXXXXX)nfnic: <1>: INFO: fnic_fdls_link_status_change: 99: fnic1: FDLS link status change link up:0, usefip:0
cpu0:XXXXXXX)nfnic: <1>: INFO: fnic_fdls_link_down: 4504: Moving tport to PLOGO state tport: 1 fcid: 0xXXXXX
cpu0:XXXXXXX)nfnic: <1>: INFO: fnic_fdls_link_down: 4504: Moving tport to PLOGO state tport: 2 fcid: 0xXXXXX
cpu0:XXXXXXX)WARNING: nfnic: <1>: fnic_fw_reset_handler: 307: Issued fw reset


### Issued fw reset indicates a firmware crash. 

yy-mm-dd In(182) vmkernel: cpu47:XXXXXXX)nfnic: <2>: INFO: fnic_taskMgmt: 2080: VMK_SCSI_TASKMGMT_ABORT issued. iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED
yy-mm-dd In(182) vmkernel: cpu47:XXXXXXX)nfnic: <1>: INFO: fnic_taskMgmt: 2080: VMK_SCSI_TASKMGMT_ABORT issued. iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED

### At the same time logs show FNIC_IPORT_IO_BLOCKED

• The number of events that are logged for FNIC_IPORT_IO_BLOCKED in the vmkernel logs are shown by using the following command at the ESXi command line:
  
  # less vmkernel.*** | grep -i FNIC_IPORT_IO_BLOCKED | wc -l
314115

Environment

7.x
8.x

Cause

• The logs show a firmware crash. This indicates an issue with the Firmware. 
• "state: 3" means FNIC_IPORT_STATE_FABRIC_DISC (fabric discovery stage).  
• FNIC_IPORT_IO_BLOCKED messages indicate that the iport is not in a ready state. Such log message could be triggered by various factors , i.e. during the Fabric Discovery Stage(state 3) , there is a burst of I/Os sent to the iport which is not ready, etc. 

Resolution

 

• Refer this KB as general guideline when upgrading I/O device firmware and driver : "Supported drivers and firmware version for I/O devices"
• Engage hardware vendor to analyze firmware dump for further investigation and recommendations.