Congestion Oversubscription and Credit Stall events reported in /var/log/vmkernel.log on ESXi Server as well as vCenter Server
Article ID: 390100
Updated On:
Products
Issue/Introduction
Events similar to below are reported in /var/log/vmkernel.log as well as vCenter Server:
2025-02-25T14:12:13.015Z In(182) vmkernel: cpu36:2098098)StorageFPIN: 1276: Report FC FPIN Congestion Oversubscription event (hostWWPN xxxxxxxxxxxxxxxx tgtWWPN yyyyyyyyyyyyyyyy) to vobd. 231 events have occurred since last report.
2025-02-23T07:50:00.079Z In(182) vmkernel: cpu45:2098118)StorageFPIN: 1276: Report FC FPIN Congestion Credit Stall event (hostWWPN xxxxxxxxxxxxxxxx tgtWWPN yyyyyyyyyyyyyyyy) to vobd. 6 events have occurred since last report.
2025-02-23T07:50:00.079Z Wa(180) vmkwarning: cpu52:[REDACTED_ID])WARNING: lpfc : vmhbaX lpfc_els_rcv_fpin_cgn:[REDACTED_ID]: [REDACTED_ID] FPIN CONGESTION WARNING Notification type Credit Stall (x2) Event Duration 10000 mSecs
2025-02-23T07:50:00.079Z In(182) vmkernel: cpu52:[REDACTED_ID])StorageFPIN: [REDACTED_ID]: Report FC FPIN Congestion Credit Stall event (hostWWPN [MASKED_WWPN] tgtWWPN [MASKED_WWPN]) to vobd. 4 events have occurred since last report.
2025-02-23T07:50:00.079Z In(182) vmkernel: cpu52:[REDACTED_ID])StoragePath: [REDACTED_ID]: Calling MPP NMP for link event 2 on adapter vmhbaX (hostWWPN=[MASKED_WWPN] targetWWPN=[MASKED_WWPN] targetNum = [MASKED_NUM])
2025-02-23T07:50:00.079Z Wa(180) vmkwarning: cpu52:[REDACTED_ID])WARNING: NMP: nmpHandleLinkEvent:[REDACTED_ID]: Marking path vmhbaX:C0:T0:L00 flaky on link event 2 with timeoutMS = 20000 flakyMarkTC = [MASKED_ID], reEvalFlakyPathTime = 20000
Cause
FPIN (Fabric Performance Impact Notifications) capability was added in ESXi 8.0 U2 to be able to better understand fabric related issues/events. This module will also print to /var/log/vmkernel.log when there are fabric events happening. The events that FPIN tracks and will report on are:
- Link Integrity
- Delivery
- Congestion
- Peer Congestion
Resolution
When FPIN events are received, the fabric health of the switching fabric should immediately be investigated. For the example listed above, this initiator is receiving FPINs that points to both Congestion as well as a Credit Stall. A Credit Stall is reported when the B2B credits for an ISL connection is reaching zero, which typically happens when there is a Slow Drain device in the fabric. Regardless, the presence of FPIN events indicates that the fabric should immediately be investigated for root cause otherwise the fabric health could continue to reduce and an outage could occur.
For more information on Credit Stalls, please refer to the following documentation: Credit Stalls
Feedback
