Error: iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED on Cisco UCS

Products

• VMware vSphere ESXi
• VMware Cloud Foundation
• VMware Telco Cloud Platform

Issue/Introduction

• Host status changes to "Not Responding" in vCenter
• Virtual machines on the affected host become inaccessible or hung.
• VMkernel logs show high frequency of the following errors:
  

  nfnic: <2>: INFO: fnic_taskMgmt: 2080: VMK_SCSI_TASKMGMT_ABORT issued. iport not in ready state. state: 3 or FNIC_IPORT_IO_BLOCKED
WARNING: nfnic: <1>: fnic_fw_reset_handler: 307: Issued fw reset

Environment

ESX: 7.x, 8.x, 8.0 Update 3h 

VCF: 5.2.2

TCP: 5.0, 5.0.1, 5.0.2, 5.1

Cisco UCS server with nfnic driver

Cause

Cisco fnic adapter initiator port enters a blocked I/O state (state: 3) during fabric discovery and fails to recover via firmware resets, leading to a permanent loss of storage connectivity.

Resolution

1. Ensure driver and firmware combination is supported - see Supported drivers and firmware version for I/O devices
2. Determining Network/Storage firmware and driver version in ESXi
3. Engage Cisco to analyze firmware logs for further investigation and recommendations.