Error: "esx.problem.hardware.fpin.fc.congestion.oversubscription" on hosts within a vSphere cluster.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vCenter reports messages similar to the following in the host/clusters events log (From Host/Cluster Inventory view, Choose Host or Cluster -> Monitor -> Events): 
  
  

Description	Type	Date Time	Task	Target	User	Event Type ID
FPIN FC oversubscription congestion: Host WWPN ##############, target WWPN ##############.	Warning	DD/MM/YYYY, 1:25:09 PM		testhost01.testlab.local		esx.problem.hardware.fpin.fc.congestion.oversubscription

 

• The /var/run/log/vobd.log on the affected ESXi host reports similar messages: 

YYYY-MM-DDTHH:MM:59.836Z In(14) vobd[2097956]:  [HardwareCorrelator] 4976524033798us: [esx.problem.hardware.fpin.fc.congestion.oversubscription] FPIN FC oversubscription congestion: Host WWPN ##############, target WWPN ##############.

Parallelly in vobd.log, below log entries can be seen:

YYYY-MM-DDTHH:MM:59.836Z In(14) vobd[2097956]:  [vmfsCorrelator] 3805055794354us: [esx.problem.vmfs.heartbeat.timedout] 65fc521d-c5b3a81e-1bf5-########### <Datastore Name>

• You may notice latency in operations such as HBA rescans and VM performance can be degraded and IOs from the adapters will start getting aborted:
  
  YYYY-MM-DDTHH:MM:59.836Z In(182) vmkernel: cpu16:4758985)qlnativefc: vmhba4(4b:0.0): qlnativefcEhVirtualReset:3238:C0:T7:L63: VIRTUAL RESET ISSUED.
YYYY-MM-DDTHH:MM:59.836Z In(182) vmkernel: cpu16:4758985)qlnativefc: vmhba4(4b:0.0): qlnativefcEhVirtualReset:3282:qlnativefcEhVirtualReset: abortCommand mbx success.

YYYY-MM-DDTHH:MM:59.836Z Wa(180) vmkwarning: cpu66:2098586)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.################################" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM:59.836Z In(182) vmkernel: cpu66:2098586)ScsiDeviceIO: 4616: Cmd(0x45da3b0cf080) 0x8a, cmdId.initiator=0x430f263517c0 CmdSN 0x8000001e from world 4235355 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from driver

• User might observe a driver firmware mismatch on the configured FC host bus adapters.
  
  

• There can be flaky path events on vmhba interfaces similar to below under /var/run/log/vmkernel.log:

  YYYY-MM-DDTHH:MM:59.836Z Wa (180) vmkwarning: cpu50:2098673) WARNING: NMP: nmpHandleLinkEvent : 3992: Marking path vmhba4:C0:T2:L63 flaky on link event 2 with timeoutMS = 120000 flakyMarkTC = 260834006239432, reEvalFlakyPathTime = 120000

Environment

• VMware vSphere ESXi 8.x

• VMware vSphere ESXi 9.x

Cause

• The fabric ports related to the impacted hosts are overutilized. See related KB:  Congestion Oversubscription and Credit Stall events reported in /var/log/vmkernel.log on ESXi Server as well as vCenter Server
  
  
• The slowness can be caused by a combination of unstable (flaky) storage paths and SAN fabric congestion (FPIN oversubscription).
  
  
• The fabric issues will be exacerbated by the host HBA firmware not being at the recommended level, leading to inefficient I/O handling and SCSI reservation constraints. Check the same in Broadcom Compatibility Guide

 

Resolution

1. If only a small number of hosts in the cluster are affected, then, in the short term, you can try to re-distribute the VM's across the ESXi hosts in the cluster to balance the I/O across the fabric ports more evenly, possibly adding or removing DRS rules to pin or unpin VM's to/from certain hosts, as needed to keep the storage I/O load distributed. For more information regarding add/removing DRS rules, see TechDocs: Use DRS Clusters to Manage Resources with vSphere
   
   
2. When possible, try to use a Round-Robin Path Selection Policy to help distribute the load across the available HBA ports on the hosts. See KB: VMware Multipathing policies in ESXi/ESX
   
   
3. If all of the hosts/fabric switch ports are affected (implying all ports are being overwhelmed), then you will need to consider adding additional HBA's and/or ESXi hosts to the cluster to help distribute the storage I/O across more fabric ports or possibly upgrading to faster switches, if possible.
   
   
4. In any case, consult with SAN, fabric switch vendor, and Broadcom Support for recommendations for your specific environment to ensure the hardware is sized and balanced appropriately.

Additional Information

• vCenter/ESXi has a few tools to monitor VM performance to determine high I/O VM's:
  
  
  • Monitor the VM performance in the vCenter Performance Charts. For more information on using the performance charts, see: 
    • TechDocs:
      • View Performance Charts
      • Read IOPs per Virtual Machine Disk and Write IOPs Per Virtual Machine Disk
        
        
  • Monitor VM performance using the ESXi tool, esxtop. For more information on using esxtop, see:
    • TechDocs: Using esxtop or resxtop in Interactive Mode
    • KB: Using esxtop to identify storage performance issues for ESXi

• • Use VMware Aria Operations to monitor VM performance, as well.  See TechDocs: VMware Aria Operations